Systems and Methods for Autonomous Intravenous Needle Insertion

ABSTRACT

Systems and methods for autonomous intravenous needle insertion are disclosed herein. In an embodiment, a system for autonomous intravenous insertion include a robot arm, one or more sensors pivotally attached to the robot arm for gathering information about potential insertion sites in a subject arm, a medical device pivotally attached to the robot arm, and a controller in communication with the sensors and the robot arm, wherein the controller receives the information from the sensors about potential insertion sites, and the controller selects a target insertion site and directs the robot arm to insert the medical device into the target insertion site.

RELATED APPLICATIONS

This application is a divisional patent application of U.S. applicationSer. No. 13/335,700 filed on Dec. 22, 2011, which claims the benefit ofand priority to U.S. Provisional Patent Application No. 61/426,022 filedon Dec. 22, 2010, the entirety of each of which is hereby incorporatedherein by reference for the teachings therein.

FIELD

The embodiments disclosed herein relate to intravenous insertionsystems, and more particularly to autonomous intravenous insertionsystems and methods for using the system for autonomously inserting aneedle or cannula into a vessel of a patient to be treated.

BACKGROUND

Intravenous needle and cannula insertion are mainstay procedures withinmodern medicine. They are essential components of both drug delivery andblood sampling for diagnostic purposes. Despite the fact thatintravenous needle insertion, especially in forearm veins, is one of themost commonly practiced medical procedures, it is notorious for being anunmastered technique. Many patients are poked several times before theneedle is successfully inserted and there is often great variability inadeptness among medical personal with regard to their needle insertionskills. Accordingly, there is a need to automate needle insertion tolessen the dependence on skilled technicians, decrease procedure time,and to reduce errors during intravenous needle and cannula insertionprocedures.

SUMMARY

Systems and methods for autonomous intravenous needle insertion aredisclosed herein. According to aspects illustrated herein, there isprovided a system for autonomous intravenous insertion that includes arobot arm, one or more sensors pivotally attached to the robot arm forgathering information about potential insertion sites in a subject arm,a medical device pivotally attached to the robot arm, and a controllerin communication with the sensors and the robot arm, wherein thecontroller receives the information from the sensors about potentialinsertion sites, and the controller selects a target insertion site anddirects the robot arm to insert the medical device into the targetinsertion site.

According to aspects illustrated herein, there is provided a system forautonomous intravenous insertion that includes a robot arm, a pluralityof sensors attached to the robot arm for gathering information aboutpotential insertion sites in a subject arm, a medical device holdingtool detachably engaged to the robot arm, the tool comprising aplurality of grippers for holding a medical device to be inserted intothe subject arm, a first actuating mechanism for actuating the pluralitygrippers, stabilizing feet; and a second actuating mechanism for placingthe stabilizing feet in the proximity to an insertion site, and acontroller in communication with the plurality sensors, the medicaldevice holding tool, and the robot arm, wherein the controller receivesthe information from the sensors about potential insertion sites, andselects a target insertion site and directs the medical device holdingtool and the robot arm to insert the medical device into the targetinsertion site.

According to aspects illustrated herein, there is provided a method forautonomous intravenous insertion that includes securing a subject arm,identifying a target insertion site based on information received fromat least one sensor, actuating a robot arm to deliver a medical deviceto the target insertion site, while monitoring the target insertionsite, and inserting the medical device into the subject arm at theinsertion site.

BRIEF DESCRIPTION OF THE DRAWINGS

The presently disclosed embodiments will be further explained withreference to the attached drawings, wherein like structures are referredto by like numerals throughout the several views. The drawings shown arenot necessarily to scale, with emphasis instead generally being placedupon illustrating the principles of the presently disclosed embodiments.

FIG. 1 is a perspective view of an embodiment of an autonomousintravenous insertion system of the present disclosure.

FIG. 2A and FIG. 2B are a schematic illustration of an embodiment of anautonomous intravenous insertion system of the present disclosure.

FIG. 3A and FIG. 3B illustrate an embodiment of a sensor assembly foruse with an autonomous intravenous insertion system of the presentdisclosure.

FIG. 4A, FIG. 4B and FIG. 4C illustrate various medical device holdingtools for use with an autonomous intravenous insertion system of thepresent disclosure.

FIG. 5 illustrates an embodiment of a catheter tool of the presentdisclosure for holding a catheter.

FIG. 6 illustrates an embodiment of a catheter tool in its initialstate, without a catheter.

FIG. 7, FIG. 8, FIG. 9, and FIG. 10 are cutaway views of an embodimentof a catheter tool at various stages of a catheter insertion procedure.

FIG. 11 illustrates an embodiment of a needle tool of the presentdisclosure for holding a needle.

FIG. 12 illustrates an embodiment of a needle tool with its stabilizingfeet in a deployed positions.

FIG. 13 and FIG. 14 are cutaway are cutaway views of an embodiment of aneedle tool at various stages of a needle insertion procedure.

FIG. 15A, FIG. 15B and FIG. 15C show an embodiment of a gripper assemblyand linear actuator of a needle tool.

FIG. 16 shows an embodiment of a blood drawing tubes manipulator of thepresent disclosure.

FIG. 17, FIG. 18 and FIG. 19 show an embodiment of a blood drawing tubesmanipulator in operation.

FIG. 20 is a block diagram illustrating components of an embodiment ofan autonomous intravenous insertion system of the present disclosure.

FIG. 21 is a flow diagram illustrating hardware units of an embodimentof an autonomous intravenous insertion system of the present disclosure.

FIG. 22 is a schematic diagram illustrating components of an embodimentof an autonomous intravenous insertion system of the present disclosure.

FIG. 23 is a schematic of software architecture and modules in anembodiment autonomous intravenous insertion system of the presentdisclosure.

FIG. 24 is a flowchart of a method for use of an embodiment of anautonomous intravenous insertion system of the present disclosure.

FIG. 25 is a screenshot of an interface of an embodiment of anautonomous intravenous insertion system of the present disclosure.

FIG. 26 is a screenshot of an interface of an embodiment of anautonomous intravenous insertion system of the present disclosure.

FIG. 27 is a screenshot of an interface of an embodiment of anautonomous intravenous insertion system of the present disclosure.

FIG. 28 is a block diagram illustrating an internal architecture of acomputer in accordance with an embodiment of the present disclosure.

While the above-identified drawings set forth presently disclosedembodiments, other embodiments are also contemplated, as noted in thediscussion. This disclosure presents illustrative embodiments by way ofrepresentation and not limitation. Numerous other modifications andembodiments can be devised by those skilled in the art which fall withinthe scope and spirit of the principles of the presently disclosedembodiments.

DETAILED DESCRIPTION

Systems and methods for autonomous intravenous needle insertion aredisclosed herein. In an embodiment, the systems disclosed herein act asautonomous intravenous needle insertion systems for inserting a medicaldevice into a vessel of a patient during a medical procedure. In anembodiment, the systems disclosed herein act as autonomous intravenousneedle insertion systems for inserting a medical device into a vessel ofa patient during an insertion procedure. In an embodiment, the systemsdisclosed herein act as autonomous intravenous needle insertion systemsfor inserting a needle into a vein of a patient to be treated. In anembodiment, the systems disclosed herein act as autonomous intravenousneedle insertion systems for inserting a cannula into a vein of apatient to be treated. In an embodiment, the systems disclosed hereinact as an autonomous blood drawing system.

Subcutaneous vein-finding has been a subject of interest that has takenoff in the past decade. The most prominent techniques used to identifyveins thus far are infrared imaging and ultrasound signal processing.Infrared imaging is used to highlight the difference in contrast betweena vessel and the surrounding skin when the region of interest isilluminated by infrared radiation. Ultrasonic systems are used to verifythat a vein is indeed present under the surface of the skin where theultrasound probe is deployed. Ultrasound may also be used to determineflow in a vessel using Doppler Shift analysis or it may be used to imageveins underneath the skin.

The systems described herein can uniquely combine the infrared imageswith ultrasound images to highlight veins within these images based onshape, size, and orientation, among other characteristics. In anembodiment, the systems described herein are capable of selecting themost suitable vein for needle insertion based on various parametersincluding, but not limited to, location within the arm, size,orientation, and certainty of the selection being a vein. In anembodiment, the systems described herein track and localize the needleinsertion site selected in real time so as to not lose the position ofthe needle insertion cite. In an embodiment, the systems describedherein operate reliably and have built-in safety features so that thesystems only operate under the circumstances appropriate for the medicalprocedure to be performed.

The autonomous intravenous needle insertion systems of the presentdisclosure are designed to take the place of a nurse or technician inthe task of blood drawing and intravenous (IV) insertion. Other than foreveryday phlebotomy or IV insertion on active patients, the autonomousintravenous needle insertion systems of the present disclosure can beused to administer drugs to subjects through syringes. In an embodiment,the autonomous intravenous needle insertion systems described herein canaid with needle insertion procedures required during surgery. Potentialuses of the autonomous intravenous needle insertion systems of thepresent disclosure include, but are not limited to, automation ofeveryday phlebotomy procedures, aid with surgical procedures, carry outcatheter insertion into a blood vessel, as part of a mobile robot withfunctionalities of a medical technician for care of soldiers orastronauts or other hard to reach workers needing immediate medicalattention, for administering drugs to or gathering medical diagnosticsfrom highly contagious or diseased patients, or in settings where it isnot safe for medical personnel to travel, for instance as part of arescue mission in the midst of biological warfare, and in a laboratorysetting for automation of experiments involving injection.

Autonomous intravenous needle insertion systems may be advantageous fora number of reasons, including, but not limited to, minimizing thenumber of unsuccessful insertion attempts of a medical device into avessel of a patient, and to lessen the dependence on skilled techniciansand nurses in busy hospital environments as well as in the field, wheresuch skills may be otherwise unavailable. In addition to theseadvantages in a hospital or office setting, the autonomous intravenousneedle insertion systems of the present disclosure also may be utilizedin less conventional settings, including, but not limited to, researchand laboratory settings necessitating many needle insertion proceduresto be carried out simultaneously or after short intervals of time on oneor more persons, or insertion in an unsafe environment, settingsinvolving highly contagious or diseased patients, or other such settingswhere needle insertion is needed but it is unsafe for medicaltechnicians or personnel (for instance a rescue mission in the midst ofbiological warfare), and settings where it is not practical for medicaltechnicians to go, such as a battlefield, an airplane, or outer space.

Generally speaking, the medical procedure for drug delivery or bloodsampling includes the following steps: 1) gathering the appropriateapparatus for the procedure; 2) locating an insertion site by visualinspection and/or palpation; 3) carrying out the procedure; 4)discarding insertion apparatus and disposable paraphernalia that is notneeded following the procedure. The autonomous intravenous needleinsertion systems of the present disclosure are capable of accomplishingthese tasks autonomously, while mimicking human activity on many levels.The autonomous intravenous needle insertion systems of the presentdisclosure are also capable of being operated by a medical technician,nurse, or doctor, in the event they wish to do so.

The medical procedure for drug delivery or blood sampling involves morethan just inserting a needle into a patient's vessel. In the case ofdrawing blood with VACUTAINER® equipment, the procedure involvesinserting the needle, engaging the vacuum tubes, disengaging the tubesafter they are filled, and replacing a filled tube with another emptytube when multiple specimens are needed. To that end, the autonomousintravenous needle insertion systems of the present disclosure include adevice to handle tube engagement, disengagement, and new tubereengagement at the tail end of the VACUTAINER® needle device. In thecase of cannula insertion, the needle must be extracted from thein-dwelling catheter surrounding it, leaving the cannula in place toadminister drugs or to draw blood. To that end, the autonomousintravenous needle insertion systems of the present disclosure include aneedle extraction mechanism.

In addition, there are other ancillary procedures involved in needle orcannula insertion. For example, when a technician draws blood, thetechnician uses a tourniquet to build up pressure in the vessel, andwhen the technician has finished drawing blood, the tourniquet isreleased. The technician also applies pressure to the site afterremoving the needle in order to prevent bleeding. In the case of IVcannula insertion, the insertion site and catheter are dressed followinginsertion in order to prevent contamination and dislodgement of thecatheters. The autonomous intravenous needle insertion systems of thepresent disclosure are capable of handling such ancillary proceduresrelated to the medical procedure at hand. At the start of the procedure,an automatic tourniquet device serves the dual purpose of ordinarytourniquet function and as an arm stabilizer.

FIG. 1 and FIG. 2A illustrate an embodiment of an autonomous intravenousinsertion system 8 of the present disclosure, having an insertion module215 attached to a robot arm 1. In an embodiment, the insertion module215 includes a main sensor assembly 2, shown only with a near-infraredlight source 62, a butterfly needle 41 and a medical device holdingtools. Exemplary medical device holding tools include a needle tool 3for drawing blood and a catheter tool 4 (not shown) for insertingcatheters into the subject. Various tools and sensors of the insertionmodule 215 may be attached to the robot arm 1 by means known and used inthe art. In an embodiment, a tool changer system having a male toolchanger 9 and a female tool changer 10 is used for connecting medicaldevice holding tools, such as the needle tool 3 or catheter tool 4, tothe robot arm 1.

In an embodiment, the robot arm 1 is programmable to actuate medicaldevice holding tools to insert and extract needles or catheters viamedical device holding tools attached to the end of the robot arm 1. Inan embodiment, medical device holding tools of the present disclosureinclude the needle tool 3 for handling blood drawing equipment andcatheter tool 4 for catheter manipulation. The robot arm 1 also travelsto docking positions that vary according to the procedure to beperformed. Each contemplated medical device, such as a butterfly needle41, has it own automatic loading mechanisms to append the device ontothe tool designed to manipulate it. In an embodiment, the needle tool 3appends a butterfly needle 41 to its end effector autonomously with helpof an automatic butterfly needle dispenser 11. Once the butterfly needle41 is appended to the needle tool 3, the three-dimensional spatialcoordinates along with the orientation of the selected vein aredetermined, and the robot arm 1 moves the butterfly needle 41 to thatposition for insertion. After the procedure is performed, the robot arm1 travels to an unloading position in order to discharge the usedmedical device.

In an embodiment, the robot arm 1 is computer controlled and is designedto autonomously guide a medical device to a patient's vessel for theautomatic insertion of the medical device into the patient's vessel. Inan embodiment, the robot arm 1 may include a high-precision mechanicalpositioning system and a control unit. In an embodiment, the controlunit is a dedicated computer and motor driver that takes commands fromthe master computer 90 and controls the robot's position and motion. Anemergency stop switch 13 may be present in case of system malfunction,allowing immediate cessation of the motion of the robot arm 1. Themotion of the robot arm 1 may also be immediately stopped through thecontrolling software, if desired. If the robot arm 1 is stopped ineither of these ways, it may retreat to its original home position. Inan embodiment, the robot arm 1 may be provided with capability to holdits present position, while allowing someone to manipulate its jointswith ample force may also be included. In an embodiment, as anadditional safety feature, the robot arm 1 does not move as a result ofgravitational forces. In an embodiment, the robot arm 1 is a mobilerobot capable of avoiding obstacles and path following so that thesystem 8 can autonomously move around a hospital room or floor.

In an embodiment, a pneumatic system 94 is connected to the robot arm 1to provide controlled air pressure to end-effectors at the end of therobot arm 1 through tubes that reach from the base of the robot arm 1 tothe manipulator end. In an embodiment, the tool changers 9 and 10, theneedle tool 3, the catheter tool 4, and a disposal unit 96 all usepneumatic actuation. In an embodiment, the pneumatic system 94 includesa small compressor and storage tank, and a series of computer-controlledvalves used to operate various pneumatic actuators used in the endeffectors.

In an embodiment, pneumatic actuators are used to grip medical devices,such as butterfly needles 41, catheters 22, and other medical devices.In an embodiment, butterfly needles 41 or catheters 22 are attached tothe needle tool 3 or the catheter tool 4, respectively, for facilitationof the insertion procedure. In an embodiment, other devices may be usedto assist with the medical procedure to be performed in conjunction withthe insertion procedure, including vacuum tubes for blood drawing, amedication administration device, or other diagnostic or therapeuticdevices, including, but not limited to, blood drawing equipment,syringes, medication pumps, intravenous solution bags, or combinationsthereof. In an embodiment, the medical device to be manipulated is aVACUTAINER® butterfly needle 41. In an embodiment, the medical device tobe manipulated is a catheter 22. In an embodiment, the butterfly needle41 or catheter 22 may be attached to the complementary device forfacilitating a medical procedure to be performed in conjunction with theinsertion procedure.

FIG. 2B illustrates a patient's arm 7 cuffed inside an automatictourniquet cuff 5 and wrist stabilizing cuff 6, which can be used toincrease the visibility of veins and stabilize the patient's arm 7during the procedure. Both the automatic tourniquet cuff 5 and wriststabilizing cuff 6 are pressurized with air after the patient's arm 7 isplaced through the cuffs 5 and 6. In an embodiment, the automatictourniquet cuff 5 serves as an ordinary tourniquet, that is, itincreases vessel size and visibility which ultimately aids with the veinidentification procedure. In an embodiment, together with the automatictourniquet cuff 5, the wrist stabilizing cuff 6 serves as a rest and ademobilizer for the patient's forearm to simplify the tracking of thetarget vessel. In an embodiment, a barcode reader 216 is added near thewrist stabilizing cuff 6 that would read a patient's bracelet barcode inorder to collect patient data for the procedure. If not wearing awristband, patients might enter their information into the touch screenuser interface. In an embodiment, the automatic tourniquet cuff 5 andwrist stabilizing cuff 6 are positioned on a flat surface.

FIG. 3A and FIG. 3B show an embodiment of the main sensor assembly 2used to find appropriate insertion sites on the patient's arm 7. In anembodiment, the main sensor assembly 2 is permanently attached to therobot arm 1 and is not designed to be changed out routinely. In anembodiment, the main sensor assembly 2 includes one or more sensoryunits or primary sensors 91, such as a laser rangefinder 60, anear-infrared (NIR) camera 61 and a near-infrared (NIR) light source 62.The main sensor assembly 2 may also include an ultrasound device 64 (notshown in FIG. 3A or FIG. 3B). The main sensor assembly 2 may alsoinclude miniature solenoid valves 63 for control of pneumatic devicesand the male tool changer 9, which is used for connecting medical deviceholding tools to the robot.

In an embodiment, one or more of the primary sensors 91 may be rigidlymounted on the robot arm 1 in a fixed relation to the butterfly needle41. In other words, if the butterfly needle 41 pivots, the one or moreprimary sensors 91 pivot with the butterfly needle 41. In an embodiment,one or more of the primary sensors 91 may be positioned at a stationaryvantage point. In an embodiment, the one or more primary sensors 91provide positioning information to the control unit of the system 8,allowing the autonomous intravenous insertion system 8 to recognizepotential insertion sites and their locations and orientations in thecoordinate system of the robot arm 1.

One of the primary sensors 91 in the main sensor assembly 2 comprises animaging unit, such as a CMOS, CCD, or other similar camera coupled witha bandpass filter used to isolate a near-infrared (NIR) frequency range.This imaging device is referred to throughout this document as an NIRcamera 61. In an embodiment, the NIR frequency range is a narrowfrequency range. In an embodiment, the wavelength range used by thesystem to distinguish the patient's vessel from the environmentsurrounding the vessel comprises a range from about 720 nanometers toabout 780 nanometers. It should be noted that other NIR wavelengthranges may be used without departing from the spirit and scope of thepresently disclosed embodiments. It should also be noted that othermedical imaging techniques can be employed by the system to highlight apatient's vessel in contrast to the environment surrounding thepatient's vessel.

In an embodiment, the one or more primary sensors 91 comprise an NIRcamera 61 for determining the three-dimensional coordinates of thevessel of the patient receiving the medical procedure. In an embodiment,the NIR camera 61 can be used to determine an optimal orientation forinsertion of a medical device into the vessel of the patient. In anembodiment, the NIR camera 61 may provide relative location in twodimensions, as well as the orientation of the targeted vessel relativeto the current position of the butterfly needle 41 at the distal end ofthe robot arm 1, so that the butterfly needle 41 may be aligned with thetargeted vessel to ensure a safe insertion procedure.

In an embodiment, the autonomous intravenous insertion system 8 may usemultiple NIR cameras 61 capable of seeing different frequency ranges inorder to better eliminate false vein images from appearing to be validones. In an embodiment, multiple NIR cameras 61 can provide a grayscalecolor image and a NIR image, which can be stitched together, and pixelvalues of target sites in the NIR image can be compared to correspondingpixels in the grayscale color image. Because NIR-only cameras pick upvein contrast better than white light cameras, it is expected that thecomparison of the two images containing different frequencyrepresentations of the insertion region may lead to an image with datacontaining information about the location of potential veins. Morespecifically, true veins may have substantially darker pixel values inthe NIR image than they will in the grayscale white light image.Analysis of these comparisons can lead to less false positives andtherefore more accurate target site selection.

In an embodiment, the one or more primary sensors 91 comprise a laserrangefinder 60 to determine the distance of the butterfly needle 41 tothe patient's vessel. In an embodiment, the laser rangefinder 60 can beused to determine the topography of the selected insertion site. Thelaser rangefinder 60 can be mounted on the robot arm 1 in a fixedrelation to the insertion needle. In an embodiment, the laserrangefinder 60 is designed to operate in conjunction with the NIR camera61 to track the three-dimensional coordinates and orientation of apatient's vessel in real-time to identify an optimal insertion path forinserting a medical device into the vessel. In an embodiment, the laserrangefinder 60 is designed to operate in conjunction with the NIR camera61 to enable the robot arm 1 of the system 8 to guide a medical deviceattached to the robot arm 1 along the optimal insertion path toautonomously insert the medical device into the vessel. In anembodiment, the laser rangefinder 60 and the NIR camera 61 work togetherto determine the three-dimensional coordinates and orientation of thepatient's vessel so that the primary actuators 93 can guide the medicaldevice to be inserted into the patient's vessel to the three-dimensionalcoordinates of the vessel for insertion of the medical device into thevessel.

In an embodiment, the laser rangefinder 60 comprises a high-precisionlaser rangefinder. The laser rangefinder 60 uses a laser beam todetermine the distance to the vessel. The laser rangefinder 60determines the distance to the vessel by sending a laser pulse in a beamtowards the vessel and measures the time it takes for the pulse to bereflected off of the vessel and for the pulse to return to therangefinder. In an embodiment, the laser rangefinder 60 can be used todetermine other distances useful for determining the three-dimensionalcoordinates of the vessel or the orientation of the vessel, including,but not limited to the distance to the patient's arm 7, for example. Thedistance to the target the laser rangefinder 60 is pointing at (the zcoordinate in the robot frame of reference) enables the user todetermine the x and y coordinates in the robot frame of that same targetas identified in the two-dimensional video frame from the camera NIR 61.Thus, the user can be provided with a three-dimensional point in spacewhereby the user can direct the movement of the robot arm 1 to a desiredsite. In an embodiment, the laser rangefinder 60 is also used to obtainthe relative topography of the insertion site, so the robot arm 1positions itself in a way that the insertion path is not obstructed inany way by the patient's arm 7.

In an embodiment, the primary sensors 91 further include an ultrasounddevice 64 (having one or more ultrasound transducers) for acquiring anultrasound image of the vessel of the patient receiving the medicalprocedure. In an embodiment, the ultrasound device 64 can be manuallyoperated by the user of the autonomous intravenous insertion system 8 ofthe present disclosure to acquire the ultrasound image of the patient'svessel to verify the existence of the patient's vessel before insertionof the medical device into the patient's vessel. In an embodiment, theultrasound device 64 can be integral with and operated autonomously bythe system 8 to acquire the ultrasound image of the patient's vessel toverify the vessel's existence before the medical device is inserted intothe patient's vessel by the actuator.

In an embodiment, the ultrasound device 64 may serve as the primaryimaging unit in a case where the system 8 has determined that thepatient has obscure veins. In this case, the ultrasound device 64 may bedeployed to image veins and determine a location of a suitable insertionsite so that the autonomous intravenous insertion system 8 can commandthe actuator to insert a butterfly needle 41 in that site. In anembodiment, the ultrasound device 64 can be manipulated by a separateultrasound manipulation unit that is appended to the robot arm 1. In anembodiment, the ultrasound device 64 can be built into the needle tool 3or catheter tool 4.

In an embodiment, the autonomous intravenous insertion system 8 may,among other functions, 1) autonomously handle and use an ultrasounddevice 64 to verify that a target is indeed a vein as well as to track avein's position underneath the skin, 2) use haptic feedback for safelyhandling the ultrasound device 64 on the subject's skin, 3) use hapticfeedback to “feel” the presence of a vessel underneath the skin, 4) usehaptic feedback to distinguish between a successful and an unsuccessfulneedle penetration, incorporate a blood pressure and temperaturemeasurement system to give the robot more “nurse-like” functionality, 5)include a NIR camera 61 aimed at the patient to monitor their reactionto the medical procedure, in case the patient has an unexpectedreaction, may enable the user to have direct control of robot arm 1movement without manually moving it such as by tele-operation of therobot arm 1, or any combination thereof.

In an embodiment, the autonomous intravenous insertion system 8 may beprovided with the capability to enable the ultrasound device 64 combinedwith haptic feedback to be autonomously or remotely manipulated. Usefulfeatures are tracking targeted tissue underneath the skin's surface inreal time, following a catheter 22 path along the body, or followingsome other medical tool being used underneath the patient's skin. Thistechnology may be considered helpful in many cases, such as in aiding adoctor or other medical personnel for (1) catheter guidance, (2) visualenhancement of surgical tool use underneath the skin, (3) trackingtissues to better aim radiation equipment during radiation therapy.

The one or more primary sensors 91 may be used in the presentconfiguration to detect the presence of subcutaneous veins underneaththe skin. The underlying purpose of these primary sensors 91 is toobtain the whereabouts and orientation of a targeted vessel as quicklyas possible. To that end, various embodiments of the autonomousintravenous insertion system 8 of the present disclosure may includemore than three units as described above and/or other sensory units suchas various near-infrared or red laser diodes, or some other handheldvein-imaging system known in the art in addition or instead of the onesdescribed above to reliably detect the presence and orientation oftargeted vessels underneath the skin.

In an embodiment, the present disclosure includes a means of controllingambient light. Lighting detection, in an embodiment, may be carried outby a combination of (1) image brightness analysis, (2) IRphototransistors, and (3) photodetectors. The lighting system includes anumber of infrared LEDs and/or a shielder for blocking room lights frominterfering with the NIR camera 61. The software can control camerasettings in combination with the brightness of the NIR LEDs to preventthe presence of light or dark spots. Diffuse lighting is imperative foreliminating shadows generating an accurate image of veins in thepatient's arm 7. This system will be controlled by the master computer90 based on the video input from the NIR camera 61, so as not to confuseor complicate the operation of the system.

FIG. 4A depicts the robot arm 1 and FIG. 4B and FIG. 4C depict twomedical device holding tools 212, needle tool 3 and catheter tool 4.These tools can be attached to the robot arm 1 using the tool changersystems 9 and 10. Needle tool 3 is shown for manipulating butterflyneedles 41, while the catheter tool 4 is used for inserting catheters22. Also depicted are stabilizer feet 20 and 40, both of which are usedfor stabilizing the subject arm 7 during procedures.

In an embodiment, automatic tool changer systems 9 and 10 are fixed tothe robot arm 1, enabling the system 8 to autonomously affix tools tomanipulate end-effectors for a desired medical procedure. These toolspick up and manipulate the medical devices 22 and 41 mentioned aboveduring insertion procedures. For example, the robot arm 1 might connectto a tool designed to pick up a certain type of butterfly needle 41, asshown in FIGS. 10-14, or a certain type of catheter 22, as shown FIGS.5-9. In an embodiment, the system 8 includes a needle gripper assembly200 for holding the butterfly needle 41 in place. In an embodiment, thesystem 8 includes a catheter gripper 21 for holding the catheter 22 inplace.

FIG. 5 is a perspective view of an embodiment of the catheter tool 4.Atop the unit is the female part of the tool changer 10 which allows theprimary actuator to append this unit. In the depicted embodiment, thecatheter tool 4 holds a 2-piece catheter 22 using a custom gripperhaving a catheter gripper 21 and a gripper finger 23. In an embodiment,the catheter 22 is stationary with respect to the catheter tool 4. In anembodiment, the catheter 22 is a standard medical device including abutterfly catheter 22 a and a retractable central needle 22 b. Theretractable central needle 22 b stiffens the butterfly catheter 22 aduring insertion, and is retracted after insertion is complete. Thebutterfly catheter 22 a is held by a catheter gripper 21 while theretractable central needle 22 b is held separately by a gripper finger23 attached to a gripper body 29. The gripper finger 23 and the gripperbody 29 can be referred to collectively as the central needle gripper23, 29. In an embodiment, the gripper body 29 can be moved along a guiderail 25 by a stepper motor 24 and a lead screw 26, which also causes thegripper finger 23 to move. In an embodiment, the catheter tool 4includes stabilizer feet 20, which are shown in FIG. 4 in their resting,upright position. The stabilizer feet 20 are used for stabilizing thepatient's arm 7 during procedures.

In FIG. 6, the embodiment of the catheter tool 4 in FIG. 5 is depictedwith all grippers open and the central needle gripper 23, 29 retractedpartway. The stabilizer feet 20 are shown in their resting, uprightposition.

FIG. 7 shows a cutaway view of the embodiment of the catheter tool 4 ofFIG. 5 in its initial state. The catheter tool 4 is designed to grip acatheter 22 and to pull the retractable central needle 22 b out of thebutterfly catheter 22 a once it is inserted. The catheter tool 4 isequipped with stabilizer feet 20 to anchor the target insertion vesselwhile the insertion procedure is underway. The stabilizer feet 20 areshown in the raised position, which can be achieved by turning thestepper motor 31 to turn a lead screw 32. In an embodiment, the leadscrew 32 moves a gear rack 27 and the axle of a stabilizer foot spurgear 36 forward until the spur gear 36 reaches the end of the lead screw32, where it stops. The gear rack 27 continues moving closer to the spurgear 36, causing the spur gear 36 to rotate. Stabilizer feet 20 areconnected to the hub of the spur gear 36, so the stabilizer feet alsorotate up as spur gear 36 rotates due to the movement of the gear rack27. In FIG. 7, the spur gear 36 is all the way forward and rotated. Inan embodiment, the axle of the spur gear 36 is threaded but is notengaged on the threads of the lead screw 32. A roller type limit switch34 may be activated when the spur gear 36 needs to move backwards. FIG.7 also shows a control circuit 30, which commands the various functionsof the catheter tool 4, and a plunger type limit switch 35, which isresponsible for preventing out-of-range motion of the rear grippers 23and 29. It should be noted that other types of limit switches can beused instead of the roller type limit switch and plunger type limitswitch. In an embodiment, a pneumatic cylinder 33 controls the frontcatheter gripper 21, and the miniature pneumatic cylinders 28 control acentral needle gripper 23, 29. As shown in FIG. 7, the catheter 22 isgripped in grippers with the retractable central needle 22 b stillinside and ready for insertion.

FIG. 8 shows a cutaway view of the next step in the insertion procedureutilizing the embodiment of the catheter tool 4 of FIG. 5, where thestabilizer feet 20 drop down and apply pressure around the insertionsite. The lead screw 32 is shown turned by the stepper motor 31 suchthat the threaded gear rack 27 is moved backward. The motion of the gearrack 27 backward relative to the spur gear 36 causes the stabilizer feet20 to swing down and press on the patient's arm 7 in proximity to aninsertion site. In an embodiment, the spring in the roller type limitswitch 34 prevents the spur gear 36 from engaging the threads of thelead screw 32 until the stabilizer feet 20 begin pressing on thepatient's arm 7. Once the spur gear 36 can no longer rotate, thebackward motion of the gear rack causes the spur gear 36 to movebackward as well, where it engages the threads of the lead screw 32 andpresses the roller type limit switch 34. When the spur gear 36 isengaged on the threads of the lead screw 32, turning the lead screw 32causes the stabilizer feet 20 to move along the axis of the lead screw32.

FIG. 9 shows a cutaway view of the embodiment of the catheter tool 4 ofFIG. 5 after the catheter 22 has been inserted in a vein. The stabilizerfeet 20 are shown in the lowered position and moved backwards to matchthe forward motion of the robot arm 1 that inserts the catheter 22 intothe patient's arm 7. In an embodiment, the speed of the gear rack 27 andspur gear 36 along the lead screw 32 matches the insertion speed of therobot arm 1 (in the opposite direction), so that the stabilizer feet 20remain stationary relative to the patient's arm 7, while the butterflycatheter 22 a is inserted.

FIG. 10 shows a cutaway view of the embodiment of the catheter tool 4 ofFIG. 5 in the next step of the insertion procedure, where theretractable central needle 22 b is retracted from the butterfly catheter22 a. In an embodiment, the stepper motor 24 turns a lead screw 26,causing the threaded gripper body 29 to slide along the guide rail 25.As the gripper body 29 is being retracted, the gripper body pulls thegripper fingers 23 backwards, which in turn pulls the retractablecentral needle 22 b with it. At this point, the butterfly catheter 22 acan be released by the catheter gripper 21, and left in the patient'sarm 7 for further procedures.

FIG. 11 shows a perspective view of an embodiment of the needle tool 3.The female tool changer 10 may be attached to the top of the needle tool3 to autonomously connect the needle tools to the robot arm 1. Theneedle tool 3 is designed to insert butterfly needles 41 into thepatient's arm 7. The butterfly needle 41 may be gripped by a needlegripper assembly 200 comprising gripper fingers 42, a butterfly needlegripper body 46, a pneumatic piston 44, and butterfly needle gripperlinkages 53. The needle gripper assembly 200 is similar to the cathetergripper 21 used in relation to the catheter tool 4 as described above. Astepper motor 43 moves the butterfly needle gripper body 46 along guiderails 45 to insert the butterfly needle 41. The needle tool 3 alsoincludes stabilizer feet 40, which are shown in FIG. 11 in theirretracted position.

FIG. 12 shows a perspective view of the embodiment of the needle tool 3of FIG. 11 after the butterfly needle 41 has been inserted. To insertthe butterfly needle 41, the stepper motor 43 pushes the needle gripperassembly 200 forward, thus inserting the butterfly needle 41 into atarget vein. FIG. 12 shows stabilizer feet 40 engaged, applying lightpressure around the insertion site to stabilize the area around theinsertion site.

FIG. 13 shows a cutaway view of the embodiment of the needle tool 3 ofFIG. 10 prior to insertion. In an embodiment, a blood draw tool controlcircuit 47 is used to command the various functions of the needle tool3. Stabilizer feet 40 are shown in the raised or upright (resting)position, which is determined by a reverse acting pneumatic cylinder 50,which is also shown in the retracted position. The needle gripperassembly 200 is shown gripping the butterfly needle 41 in a retractedposition along the guide rails 45. In an embodiment, a plunger typereverse limit switches 48 and a plunger type forward limit switch 49 areused along with a simple limit switch linkage 51 to prevent the steppermotor 43 from moving the needle gripper assembly 200 too far distally.Both limit switches 48 and 49 are shown in an inactive position in FIG.13. It should be noted that other types of limit switches can be usedinstead of the plunger type forward and plunger type reverse limitswitches.

FIG. 14 shows a cutaway view of the embodiment of the needle tool 3 ofFIG. 11 in the inserted state. Pneumatic cylinder 50 is shown engaged,pressing the stabilizer feet 40 onto the surface of the patient's arm 7.The butterfly needle 41 has been inserted into the patient's arm 7 byturning the lead screw 52 with the stepper motor 43, driving the needlegripper assembly 200 forward along guide rails 45. The plunger typeforward limit switch 49 is shown depressed in FIG. 14, preventing anyfurther insertion of the butterfly needle 41 into the patient's arm 7.

FIG. 15A, FIG. 15B and FIG. 15C show an embodiment of the needle gripperassembly 200 and linear actuator for the needle tool 3 of FIG. 11. In anembodiment, the needle gripper assembly 200 includes two butterflyneedle gripper fingers 42, two butterfly needle gripper linkages 53, apneumatic piston 44, and the butterfly needle gripper body 46. To gripthe butterfly needle 41, air pressure may be applied behind thepneumatic piston 44 inside the gripper body 46, causing the butterflyneedle gripper linkages 53 to spread the backs of the butterfly needlegripper fingers 42, closing them on the butterfly needle 41. The needlegripper assembly 200 is moveable by the stepper motor 43 along the axisof the butterfly needle 41. The stepper motor 43 turns the lead screw52, which is threaded into the gripper body 46. As the lead screw 52turns, the gripper body 46 slides along the guide rails 45, moving thegripper fingers 42 and the butterfly needle 41 with it for insertioninto the patient's arm 7.

In reference to FIG. 16, in an embodiment, the autonomous intravenousinsertion system 8 has an automatic dispenser unit 74 that engages,disengages, and exchanges blood drawing tubes 78 with the insertedbutterfly needle 41. In an embodiment, the blood drawing tubes 78 areVACUTAINER® tubes. FIG. 16 shows an embodiment of a manipulator 210 forblood drawing tubes 78. In an embodiment, the manipulator 210 is usedfor connecting the butterfly needles 41 to blood drawing tubes 78 tocollect the blood samples from the patient. In an embodiment, themanipulator 210 includes a carriage 70 sliding along a guide rail 76,propelled by the rotation of a lead screw 77. In an embodiment, a colletchuck 71 for gripping blood drawing tubes 78 is rotatably attached tothe carriage 70. In an embodiment, a storage unit 73 for filled blooddrawing tubes 78 is located below the carriage 70. A dispenser unit 74may be located next to the carriage 70. In an embodiment, the dispenserunit 74 uses a dispenser arm 75 to push new blood drawing tubes 78 intothe collet chuck 71. In an embodiment, new blood drawing tubes 78 arerotated around to the piercing needle 72, and pressed onto the piercingneedle 72, allowing blood to flow from the butterfly needle 41 through atube to the piercing needle 72, and into the blood drawing tube 78.

By way of a non-limiting example, FIG. 17 shows the manipulator 210operating to grab a blood drawing tube 78. As shown in FIG. 17, thecarriage 70 is positioned next to the dispenser unit 74 to accept a newblood drawing tube 78, and the collet chuck 71 is rotated to line upwith a blood drawing tube 78 in a dispensing position inside thedispenser unit 74. Next, the dispenser arm 75 of the dispenser unit 74pushes the blood drawing tube 78 into the collet chuck 71, whichsecurely accepts the blood drawing tube 78.

By way of a non-limiting example, FIG. 18 shows the manipulator 210operating to engage the blood drawing tube 78 with the piercing needle72. As shown in FIG. 18, the collet chuck 71 is rotated to line up withthe piercing needle 72, and the carriage 70 is driven by the rotation ofthe lead screw 77 to push the blood drawing tube 78 onto the piercingneedle 72. Contemporaneously, within the dispenser unit 74, thedispenser arm 75 is retracted to allow a new blood drawing tube 78 tomove into the dispensing position.

By way of a non-limiting example, FIG. 19 shows the manipulator 210operating to place a filled blood drawing tube 78 into the tube storageunit 73. As shown in FIG. 19, the carriage 70 pulls the filled blooddrawing tube 78 off of the piercing needle 72, and the collet chuck 71rotates down to point into the tube storage unit 73. At this point thecollet chuck 71 can drop the filled blood drawing tube 78 into thestorage unit 73 where the filled blood drawing tube 78 is kept until itcan be analyzed. Once the filled blood drawing tube 78 is released, theprocess described in reference to FIG. 17, FIG. 18 and FIG. 19 can berepeated to pick up additional blood drawing tubes 78 from the dispenserunit 74, push the blood drawing tubes 78 onto the piercing needle 72 andthen transfer filled blood drawing tubes 78 into the storage unit 78 toobtain a desired number of samples.

In an embodiment, the autonomous intravenous insertion system 8 isprovided with dispenser units 74 for needles 41, catheters 22, and blooddrawing tubes 78. In an embodiment, the system 8 is capable ofautonomous selection of correct medical devices to match the prescribedprocedure in a timely manner, and to reliably and repeatably pick up thedevices. The dispensers may also prevent contamination of the devices bykeeping them separate from the hospital environment until ready for use.The dispensers are meant to be stationed near the robot arm 1, so thatthe robot arm 1 can pick up the appropriate tool upon command withouthaving to be moved. Each piece of equipment may also be provided withits own storage unit that can be easily reloaded. When equipment is low,the user may be notified that it is necessary to renew the supply stock.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure may utilize commercially available needles 41 andcatheters 22 including, but not limited to, VACUTAINER® brand needlesand vacuum blood specimen tubes and an indwelling intravenous cannulathat can serve as a medication delivery device or a blood drawingdevice. Needles and cannulas and other devices that can be inserted intoa vessel are referred here as end-effectors since they are the terminalattachment to the robot arm and the only part of the system which isinvasive.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure is configured to handle end-effectors and othersingle use components, such as blood drawing tubes, of various designs.In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure is configured to autonomously load and disposeend-effectors and other single use components. In an embodiment, theautonomous intravenous insertion system 8 of the present disclosure isprovided with syringe manipulators, arterial catheter insertionmanipulators, ultrasound probes, or devices for other injection methods,including intradermal, subcutaneous, intramuscular or intraosseousinjection.

A storage unit 73 for filled blood collection tubes 78 may also beincluded in the autonomous intravenous insertion system 8 of the presentdisclosure to enable controlling the temperature of test sample untilthey can be analyzed. In an embodiment, such a storage unit 73 may alsokeep the test samples safe from contamination and misplacement untilthey are to be collected and tested.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure includes a disposal unit 96 for containing usedbutterfly needles 41 and catheters 22 after use. The used items in thedisposal unit 96 can be emptied daily along with other medical waste, soused items need not be handled by nurses or patients, eliminating therisk of accidental punctures with contaminated needles 41 and catheters22.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure includes a tool docking and storage station 95 forthe tools used by the system 8. In an embodiment, the tool docking andstorage station 95 is an area out of reach to patients where the robotarm 1 can affix the correct tools for the job, and store unneeded toolsuntil they are required. In an embodiment, the different devices andtools may be stored in a set of cubby-like units, where the system ispre-programmed to know what device is stored in each unit, as well ashow to pick up, affix, and manipulate it. Alternatively, anotherpossible configuration may be for each tool and complementary device tohave its own dock and storage station.

In an embodiment, the medical procedure comprises an insertionprocedure. In an embodiment, insertion procedure comprises anintravenous insertion procedure. In an embodiment, the intravenousinsertion procedure comprises intravenous insertion of a butterflyneedle 41 into the vessel of a patient's arm 7. In an embodiment, theintravenous insertion procedure comprises intravenous insertion of acannula into the vessel of the patient's arm 7. In an embodiment, thevessel comprises a vein. In an embodiment, the vessel comprises asubcutaneous vein. In an embodiment, the vessel comprises a superficialvein. In an embodiment, the vessel comprises a deep vein.

FIG. 20 illustrates a schematic diagram of an embodiment of theautonomous intravenous insertion system 8 of the present disclosure. Thesystem 8 includes primary sensors 91, including, but not limited to, alaser rangefinder 60, an NIR camera 61, and an ultrasound device 64 tocollect sensor data 92 about the patient, which is sent to a mastercomputer 90 and analyzed. Based on this sensor data 92 and the mode ofoperation, the master computer 90 sends commands to several primaryactuators 93. In an embodiment, the primary actuators 93 include anautomatic butterfly needle loader 11, an automatic catheter loader 12,needle tool 3, catheter tool 4, dispenser unit 74, as well as the robotarm 1. In an embodiment, the primary sensors 91 provide feedback as theprimary actuators 93 execute their commands, allowing for more preciseand error free operation. Feedback may include, but is not limited to,updated target position information, vein verification information,robot motion verification, ambient light measurement, and user errorssuch as excessive arm movement. Feedback is used throughout the processto ensure safe and reliable operation.

As illustrated in FIG. 20, in an embodiment, the autonomous intravenousinsertion system 8 of the present disclosure includes one or moreprimary sensors 91 for acquiring sensor data 92 in real-time relating tothree-dimensional coordinates and an orientation of a patient's veinlocated beneath the skin surface of the patient's arm 7; one or moreprimary actuators 93 for autonomous insertion of a butterfly needle 41or cannula into the patient's vein; and a master computer 90 configuredto execute a program designed to transform the acquiredthree-dimensional coordinates and orientation of the patient's vein intoan optimal insertion path for the butterfly needle 41 or cannula to beinserted into the patient's vein, wherein one or more primary sensors 91track the optimal insertion site for the butterfly needle 41 or cannulain real-time to provide a continually updated optimal insertion path sothat the master computer 90 can generate a command to the actuators toinstruct the actuators to guide the butterfly needle 41 or cannula alongthe continually updated optimal insertion path to insert the butterflyneedle 41 or cannula into the patient's vein.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure uses one or more primary sensors 91 to collectreal-time sensor data 92 about the patient. The real-time sensor data 92is sent to a master computer 90 configured to analyze the real-timesensor data 92 input into the computer. In an embodiment, the mastercomputer 90 gathers sensor data 92 from the primary sensors 91 and anysecondary sensors 213, and uses the information to coordinate theprimary actuators 93. Suitable secondary sensors 213 include, but arenot limited to, sensors that provide feedback to the primary actuators93 such as limit switches or current sensors, and other peripheralsensors like a barcode reader 216. In an embodiment, the secondarysensors 213 include a tactile sensor for sensing needle contact, a forcesensor for ‘feeling’ a successful vein penetration, or a color imagerfor detecting blood flow in the cannula.

The master computer 90, based on the analysis of the real-time sensordata 92 input into the master computer 90, outputs commands to theprimary actuators 93. In an embodiment, the primary sensors 91 andsecondary sensors 213 provide real-time feedback as the primaryactuators 93 execute their commands, allowing for precise andsubstantially error free operation of the system while autonomouslycompleting a medical procedure.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure includes software, hardware and tools to carry outany procedure in these modes, including picking up the appropriatemedical device, visualizing target insertion sites, localizing targetedinsertion sites, calculating and following a path to the targetedinsertion site, inserting the device in the appropriate location withoutharm to the patient, executing ancillary tasks that complement theprocedures, extracting the device without harm to the patient, andfinally discarding the used device are built in to the system. That isto say that the autonomous intravenous insertion system 8 needs noassistance from the user, unless so desired. In an embodiment, a usermay assist in selecting the procedure needed to be done on the patient,and replace stock equipment when necessary. The following hardware unitsenable the system to have these capabilities.

FIG. 21 depicts an embodiment layout of hardware involved in theoperation of the autonomous intravenous insertion system 8 of thepresent disclosure. For the purposes of this description, arrowspointing to a module indicate that the module the arrow points fromeither manipulates that module, or is a control input to that module.Bidirectional arrows indicate that both modules interact in some way. Inan embodiment, the main components are connected in the center of thedrawing, including the master computer 90, graphical user interface(GUI) 120, robot arm 1, and robot arm controller 1 a, the primarysensors 91, and medical device tools 212. Suitable medical device tools212 include, but are not limited to, a catheter tool 4, needle tool 3,syringe manipulator and possibly other tools 80. In an embodiment, theprimary sensors 91 and the medical device holding tools 210 are mountedonto the sixth axis of the robot arm 1, together making the robot arm'send effector. As described above, in an embodiment, the autonomousintravenous insertion system 8 includes support or secondary systems(drawn on the left side of the diagram), which aid the autonomousoperation of the autonomous intravenous insertion system 8. In anembodiment, these support systems are controlled by the master computer90. Exemplary support systems include, but are not limited to, lightingcontrol systems 61 and 62, a tool docking and storage station 95, anautomated butterfly needle dispenser unit 11, a dispenser unit 74, anautomated catheter dispenser unit 12, an automated tourniquet cuff 5, awrist stabilizing cuff 6, and a disposal unit 96.

In an embodiment, the robot arm 1 is the primary actuator in the system8, allowing for precise spatial control of the end-effector, medicaldevice holding tools 212, and/or primary sensors 91. In an embodiment,the robot arm 1 is supported by a pneumatic system 94 for powering someof the primary actuators 93 in the end-effectors and the tool changers 9and 10, and an emergency stop switch 13 for safety. The robot arm 1autonomously connects to one or more medical device tools 3 and 4 viathe tool changers 9 and 10, allowing for a wide variety of procedures tobe completed without the need for modification by a technician. Suchmedical devices include catheters, blood drawing needles and tubes, andsyringes.

In an embodiment, the autonomous intravenous insertion system 8 issupported by additional devices which allow the system 8 to functionautonomously. In an embodiment, a circuit for measuring and compensatingfor ambient light conditions 61 and 62 ensures the proper illuminationfor the NIR camera 61 functions. In an embodiment, an automatic needledispenser 11, automatic catheter dispenser 12, blood drawing tubes 78and possibly other devices used for the various procedures are includedin addition to a disposal unit 96 for used devices, allowing proceduresto be completed without the need for human contact with sterile devicesor contaminated “sharps.” In an embodiment, a storage unit 73 is alsoavailable for holding filled blood drawing tubes 78 out of harm's wayuntil the blood can be analyzed. To increase the visibility of thesubject's vessels, an automatic tourniquet cuff 5 is used. In anembodiment, the wrist stabilizing cuff immobilizes the patient's arm 7to prevent movement of the patient's arm 7 during the procedure. In anembodiment, a tool docking and storage station 95 for holding unusedend-effectors is included so that all tools are available to the system8 when the system 8 needs such tools for a procedure.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure may include three separate subsystems. In anembodiment, the first or core sub-system includes a robotic arm 1, arobot arm controller 1 a, a master computer 90, and a graphical userinterface (GUI) 120. In an embodiment, the second of the end effectorsub-system includes primary sensors 91, tool changers 9 and 10, andmedical device holding tools 212 so the robot arm 1 can operate in anygiven procedure. The third subsystem is hardware meant to take care ofall ancillary procedures that ensure the system 8 will work properly.The third subsystem modules include, but is not limited to, ambientlighting control systems 61 and 62, medical device dispensing units 11and 12, a disposal unit 96, a storage unit 73 for medical devices andother disposable devices used to carry out the procedure, an arm supportstation, a tourniquet cuff 5 or tourniquet-like applier, and a tooldocking and storage station 95, which holds tools used to grip ormanipulate medical devices used during one of the contemplated medicalprocedures.

A user interface may include a screen displaying a graphical userinterface (GUI) 120, where procedures and options will be displayed tothe user so that they may indicate what specifically the robot needs todo. User input may be coordinated with operation of the robotic arm bythe main program running on the master computer 90. The user mayinstruct this program to perform a medical procedure on the patient,such as drawing blood, inserting an IV, or delivering medicine via asyringe. Target vein selection may be carried out in one of two modesautomatic and semi-automatic. The automatic mode corresponds to thesystem running completely on its own. In such a mode, the user selectsthe procedure that needs to be done, and then leaves the system to doits job. A target insertion site will automatically be chosen by thesystem 8. The semi-automatic mode corresponds to the system highlightingpotential insertion sites based on the same insertion site-findingalgorithms used in automatic mode, and then returning these sites to theuser in highly distinguishable bounding boxes. In such a mode, the usermay be prompted to click on the corresponding box, at which time theinstant system will carry out the desired procedure on that site. If nosuitable sites are presented to the user or the user wishes to choose asite on his or her own, the user can click and drag on the desiredtarget in one of the two images presented to them by the GUI 120. Oncethe insertion location is picked, the robot operates autonomously. Ifadditional safety measures are needed, the robot may ask the user toverify the insertion site once it has reached the site's vicinity. Theuser interface provides a control panel that allows the user to choosethe procedure the robot is to carry out. It should be noted that theuser interface need not be in the vicinity of the robot; in anembodiment, it may be in a remote location.

In an embodiment, the GUI 120 is displayed on a touch screen showingvideo of either the original or processed image, modified withsuggestions target insertion sites. Once a target has been selected, theGUI 120 displays the current image from the NIR camera 61 with thetracked site highlighted. In an embodiment, the GUI 120 also gathers anyuser input that is required, for instance user verification that adesired procedure can be carried out or needs to be terminated. In anembodiment, the GUI 120 is supported by the master computer 90.

The master computer 90 coordinates all the input information from theprimary sensors 91, secondary sensors 213, if present, and the GUI 120,and also controls the actuation of the primary actuators 93, includingthe robot arm 1 and any relevant medical device holding tools 212 forhandling standard medical devices such as cannulas, Y-type catheters,VACUTAINER® phlebotomy equipment, and syringes. In an embodiment, themaster computer 90 provides a control link to the robot arm controller 1a (described below as the slave unit), which takes care of sendingsignals to the robot arm 1 to enable the movement of the robot arm 1.

By way of a non-limiting example, the master computer 90 runs the mainprogram 115 that provides a GUI 120 to the system, gathers and analyzesdata from the primary sensors 91, highlights potential insertion sites,analyzes these sights to find the best insertion location based on aunique vein classifier, localizes the targeted insertion site, tracksthe targeted insertion site, generates targeted points in space to movethrough along the targeted insertion path, commands the robot armcontrol unit with high level commands such as insert, extract, pick up,discard, change tools, etc., and receives information from the robot arm1 such as its current position in six dimensions, and its status (error,carrying out task, task done, etc). The main program 115 may communicateto the slave wirelessly or via an Ethernet port.

The master-slave system may be configured in a number of ways. Themaster and slave may be (1) rolled around together on a cart from roomto room with the robot above them, (2) be stationed in a room wireddirectly to the robot, (3) be connected wirelessly through a network, inwhich case the slave is paired with the robot, while the master computeris stationed remotely where the user operates it, or (4) both be builtin to a manually-operated or autonomous mobile robot that hascapabilities similar to the system described. Other configurations mayalso be possible.

FIG. 22 is a broad overview of an embodiment of software modules forgoverning the operations of the autonomous intravenous insertion system8. The system 8 includes a vision system 110. In an embodiment, thevision system 110 is largely responsible for decisions made by themotion control decision engine 111 in controlling the primary actuators93 involved in the system 8. In an embodiment, the vision system 110finds potential targets through a series of image processing and veinidentification algorithms. The information from the vision system 110 isone of the primary sources of information for the motion controldecision engine 111, which is responsible for controlling the primaryactuators 93 involved in the system. In an embodiment, the motioncontrol decision engine 111 further relies on support systems, such asthe ultrasound device 64 and ultrasound processor and vein verificationsystem 65 for verifying that a vein lies under the specified target,other secondary sensors 213, and possibly a barcode reader 216 forpatient identification and tube organization. It should be noted thatone or all of the support systems may be incorporated into the visionsystem 110 or one or more of the modules shown as part of the visionsystem 110 in FIG. 22 may be used as support systems.

In an embodiment, the frame grabber 110 a is responsible forcontinuously grabbing images from the NIR camera 61. The rangefinderreader 110 c associates a depth reading with the center of the image.The frame grabber 110 a feeds video to the frame processor 110 b as wellas to the GUI 120. The frame processor 110 b also feeds into the GUI120, so that the user may view both the original and the processedimages. The frame processor 110 b is responsible for processing theoriginal input image from the frame grabber 110 a in order to selectveins from the surrounding skin. The processed image is fed into thevein identification system 110 d, which includes, but is not limited to,algorithms to find the patient's arm 7, find the patient's elbow, andanalyze vein-like structures inside the arm and elbow area, amongothers.

The vein identification system 110 d feeds potential veins to betargeted for insertion to the target site selection system 110 e, which,depending on the mode of operation, enables final target selection. Thetarget site selection system 110 e can operate in a semi-automatic orfully automatic mode. In semi-automatic mode, the target site selectionsystem 110 e can suggest to the user the potential target veins that itscores best for medical device insertion. In fully automatic mode, thetarget site selection system 110 e automatically selects the best targetby beginning to track the target vein that earned the highest score outof all other scored sites. Scoring of a potential target site is basedon its size, shape, and closeness to the patient's elbow (without beingdirectly on it), and is handled within the target site selection system110 e. In an embodiment, once the target for insertion is selected, itsimage is passed to the vein tracker 110 f, which tracks the insertionsite through movements of the patient's arm 7. Tracking the target sitethrough subject movement ensures that the insertion site originallyselected remains the one that is penetrated by the medical device. In anembodiment, once tracking has begun, the motion control decision engine111 controls the robot arm 1 to approach the target site and carry outthe procedure upon arrival to the target site.

An ultrasound device 64 may be used together with the veinidentification system 110 d to detect false positives in the originalimage or to confirm true positives or negatives. In an embodiment, oneor more secondary sensors 213 may be attached to the end-effector toincrease the accuracy and reliability of the system 8 as a whole. Inanother embodiment, the secondary sensors 213 may be attached to themain sensor assembly 2 or one of the medical device holding tools 212.In an embodiment, the operation of the system 8 can also be enhanced byincluding a barcode reader 216 for scanning barcodes on the outside ofmedical devices, and a link to a database for accessing and recordingpatient data during the procedure. In an embodiment, the vision system110 may be comprised of various algorithms including but not limited toa frame grabber 110 a, a frame processor 110 b, a rangefinder reader 110c, a vein identification system 110 d, a target site selection system110 e, and a vein tracker 110 f, as shown in the top portion of FIG. 22.

In an embodiment, the first step of the vision system 110 is to have theframe grabber 110 a retrieve a frame from the NIR camera 61, and for therangefinder reader 110 c to retrieve a distance from the laserrangefinder 60. Information from these functions can be combined so thatan image always has a corresponding depth of field attached to theimage, allowing the inference of three-dimensional coordinates from atwo-dimensional image. In an embodiment, new frames are displayed to theuser on the GUI 120 along with a processed and enhanced frame thatdepicts veins in the image by highlighting them in a bright color. Atarget insertion site is selected out of the potential targets eitherautomatically or manually, as discussed above, and the target is thentracked by the vein tracker 110 f continuously until insertion iscompleted. Information from these functions can be combined so that animage always has a corresponding depth of field attached to the image,allowing the inference of three-dimensional coordinates from atwo-dimensional image. In an embodiment, new frames are displayed to theuser on the GUI 120 along with a processed and enhanced frame thatdepicts veins in the image by highlighting them in a bright color. Atarget insertion site is selected out of the potential targets eitherautomatically or manually, as discussed above, and the target is thentracked by the vein tracker 110 f continuously until insertion iscompleted.

In an embodiment, certain portions of the program controlling the system8 can run separately from the main process. These “threads” are runindependently in order to enhance their speed and reliability. In anembodiment, the frame grabber 110 a, frame processor 110 b, rangefinderreader 110 c, and vein tracker 110 f are all separate threads from themain process allowing for real-time images to be updated at regularintervals regardless of the other program functions. In this manner, ifthe main process is waiting for user input or running a time-consumingalgorithm, the frame grabber 110 a and other threads will continueoperating in the background to ensure that the information they provideis uninterrupted and up-to-date. Another advantage is that non-fatalruntime errors (such as unexpected input or other unexpectedenvironmental issues) interrupt only the main process, while any otherthreads can continue to gather data.

Video is displayed through the GUI 120, as described below in FIG. 25,FIG. 26, and FIG. 27. When target insertion sites are to be selected,live feeds are displayed to allow the user to select a site forinsertion. In an embodiment, an unprocessed feed direct from the NIRcamera 61 and a processed feed displays potential targets or trackedobjects. In an embodiment, the GUI 120 includes any necessary buttonsfor user input, for example, a manual override button in case a userwishes to choose a different vein for insertion than the selected site.

In an embodiment, the frame processor 110 b is a separate thread whichwaits for new frames from the frame grabber 110 a and processes thoseimages into a binary image (black or white, no grays). When the imagesare processed, the white parts of the image (dark spots and edges in theoriginal image) correspond to areas of interest, including withoutlimitation, veins, shadows, the sides of the patient's arm 7, or anyother objects in the frame. In contrast, black parts correspond touniform areas of similar color such as skin or background. In anembodiment, this processed image does not contain any information aboutpotential veins, but highlights all possible items in the image thatmight be veins.

The frame processor 110 b passes the processed image to the veinidentification system 110 d. In an embodiment, the vein identificationsystem 110 d identifies potential areas of interest and considers anumber of factors for each area of interest to determine if it is a veinor not, and if that vein is a viable insertion site. In an embodiment,the vein identification system 110 d is programmed so each white shapein the binary image is measured to determine if it has the correct size,shape, orientation, thickness and length to be a viable target forinsertion. In an embodiment, the vein identification system 110 d hastrained a vein classifier to take in information about the size, shape,orientation with respect to the arm, thickness, and length. All otherblobs are ignored, and the vein-like blobs are then analyzed further torank them and categorize them. In an embodiment, rankings are givenbased on the thickness of the target vessel, its overall size, andlocation relative to the subject's elbow.

If an intersection of multiple veins is encountered, such as thejunction between the basilica and median cubital vein, the intersectioncan be split into separate veins so each straight length of vein isgraded separately. Junctions can be detected by a combination ofperimeter shape analysis, convexity/concavity of the targets outercontour, perimeter-area analysis, orientation analysis, and bounding boxdimensions. In an embodiment, if the median cubital vein can becorrectly identified, it will generally rank the highest unless it isrelatively faint and another vein is deemed a better risk. If the userchooses, the user can override the selection made by the veinidentification system 110 d and choose any viable vein from theavailable options.

Once the vein identification system 110 d identifies a suitableinsertion site, the selected target is sent to the vein tracker 110 f.The vein tracker 110 f can preserve the original shape and orientationof the insertion site as selected to later compare the saved image withfuture images in order to keep the target insertion site in sightdespite of movement by the robot arm 1 and/or the subject. In anembodiment, the vein tracker 110 f updates information about therelative position of the target insertion site to the medical devicemanipulator 212 and sends coordinates and approach angles to the motioncontrol decision engine 111 to be processed into motion commands.

The motion control decision engine 111 takes the tracking informationfrom the vision system 110 and, optionally, other sensory informationfrom secondary sensors 213 or the ultrasound device 64. Based on thisinformation, the motion control decision engine 111 decides whether tomove, insert, extract, or whether it is safe to do any of these things.In an embodiment, the motion control decision engine 111 can request ascan of the target insertion site area by the laser rangefinder 60 todetermine the local topography of the patient's arm 7, so that thebutterfly needle 41 or catheter 22 is inserted in the best directionwith respect to the patient's vein and forearm. In an embodiment, therobot arm 1 is caused to move above the insertion site with visualserving techniques, based on position feedback. That is, a displacementvector is calculated from the current position to the target position,and the robot arm 1 is controlled to move along that direction. Thedistance between the insertion site and the location of the laserrangefinder 60 broken up into a path, which the robot arm 1 followsuntil the robot arm 1 reaches a position suitable to initiate insertion.In an embodiment, the robot arm 1 is positioned directly above thedesired insertion location. In an embodiment, the path for the robot arm1 is continuously updated as the image insertion site is tracked by thevein tracker 110 f, and the updated insertion path is sent to the robotcontroller 1 a subsequent to its calculation to guide the robot arm 1.

Once all the necessary information has been gathered about the insertionsite and the vein's existence has been verified, commands can be sent todeploy the stabilizer feet 20, 40 of the medical device holding tools212 so that the target vein is isolated and secured. Next, the robot arm1 may cause the end-effector to move to the final position and beinserted. In an embodiment, the motion control decision engine 111continues to monitor sensory information to ensure that it remains safeto insert the medical device tool 212 into the patient's arm 7. In anembodiment, the motion control decision engine can decide to abort theinsertion if, for example, the patient moves excessively or any of therobot functions are lost for any reason. In an embodiment, the motioncontrol decision engine 111 also selects which tools are needed for thedesired procedure, and commands the robot arm 1 to pick up selectedtools and any corresponding medical devices. In an embodiment, themotion control decision engine 111 is made aware of the correctprocedure to follow based on some form of user input, which may includedirectly using the system's GUI 120 locally or remotely, or connectingto the system via a network. In an embodiment, the system 8 is designedto work with a patient database, which can be updated to reflect thatthe patient has undergone the said procedure, along with any otherinformation required about the procedure by the particular table. In anembodiment, the database can be updated to include procedure specificnotes or details, such as how many tubes of blood were drawn or how manycc's of medicine were injected, the identification number of the medicalpersonnel who authorized the procedure, and how the patient reacted tothe procedure.

Information from the secondary sensors 213 may also be used to alter thedecision-making process. In an embodiment, a tactile sensor may beprovided to indicate a successful navigation to the user's skin. In anembodiment, a force sensor is provided to relay data about thepenetration of the butterfly needle 41 or catheter 22 through the wallof the vein.

The motion control decision engine 111 may also consider informationfrom the ultrasound device 64 when initiating or controlling theinsertion procedure. In an embodiment, the ultrasound device 64 is usedfor verification of the previously found veins or for specifying theexact location of veins underneath the skin. The ultrasound device 64may include its own software module, ultrasound processing and veinverification system 65, for decoding the complex signals from thetransducer into usable information about what is beneath the skin. Withthis system in place, an image recognition algorithm, such as templatematching, for example, may be used to identify the vein underneath theskin, find its center point, and from that find its depth underneath theskin. This depth can be sent to the robot so it can calculate where thefinal insertion position will be. Once this vein has been identified, itcan easily be tracked throughout time.

Software may also control any other medical devices or subsystems addedto the autonomous intravenous insertion system 8 of the presentdisclosure, such as a blood pressure measurement machine or atemperature measurement machine.

As mentioned above, safety precautions may be built in to the system'ssoftware. In an embodiment, the system 8 may include safety features toensure that it (1) avoids entering an infinite loop or (2) can stop theprocedure at any time. In addition, the system 8 may include an optionfor the user to extract the medical device tool 212 at any time duringthe procedure.

FIG. 23 illustrates an embodiment software architecture designed to runthe autonomous intravenous insertion system 8 of the present disclosure.The autonomous intravenous insertion system 8 includes an NIR camera 61and a laser rangefinder 60, a set of robot control instructions, and avisual control panel which enables the user to interact with the system.Video may be processed, and a method of vein selection may be chosen, atwhich time the selected area is tracked and localized. Followinginsertion site localization, further control commands may be sent to therobot control unit, such as a command for inserting the attached medicaldevice. When commands are received from the robot control unit, the unitmay return a signal to the master program verifying that the correctcommand was received.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure is controlled by a main software program 115executable on a computer configured to analyze one or more input signalsacquired by one or more sensory devices. The master computer 90 analyzesthe one or more input signals using the main program 115 and transformsthe data or information acquired by the signals into a plurality ofcommands which are then output to one or more primary actuators 93 asinstructions to then instruct the one or more primary actuators 93 toperform an intravenous insertion procedure. In an embodiment, thesignals acquired using the one or more sensory devices 91 and 213 aretransformed into images which can be viewed on a video display 121 forthe operator of the system to see. In an embodiment, the signals capturea preprocessed image and present the preprocessed image on the videodisplay 121 for the operator of the system to see. In an embodiment, themaster computer 90 is configured to transform the preprocessed image ofthe patient's arm 7 into the patient arm having a plurality of boxesprojected onto it. The plurality of boxes projected onto the patient'sarm 7 on the display function as bounding boxes to establish a perimeterwithin which a vessel is believed to exist in the patient's arm 7underneath the subject's skin. In an embodiment, the preprocessed imageis transformed using one or more image processing techniques on themaster computer 90 to a processed image. Processing of the preprocessedimage transforms an external image of a patient's arm 7 showing on theskin surface target insertion sites where a predicted vessel of thepatient is believed to exist into a processed image of the patient's arm7 showing a location of one or more actual vessels in contrast to theenvironment surrounding the patient's actual vessels.

In an embodiment, the main software program 115 may provide twofunctionalities. The first functionality is to provide a GUI 120 throughwhich the user may interact with the system 8. In an embodiment, thisentails use of the GUI 120 to control what procedure the system 8carries out, along with certain parameters to be selected to furtherdescribe the details of the procedure. The GUI 120 may also enable theuser to oversee the entire insertion process, from real-time monitoringof the medical procedure, to highlighting the targeted insertionlocation, to providing the user with the option to abort the procedureat any time. The second functionality is to integrate the system'ssensors with its state machine. The state machine (the motion controldecision engine 111) determines how the robot arm 1 will act at thecurrent time in the procedure. FIG. 24 depicts a flowchart of variousstates within the state machine, which is described in further detailbelow.

In an embodiment, as shown in FIG. 23, the system 8 includes a masterprogram 115 having a plurality of modules and a slave control program114 functioning as a robot controller 1 a. In an embodiment, the program115 is configured to transform real-time sensor data 92 acquired about alocation of a patient's vessel into a command to instruct one or moreprimary actuators 93 to execute autonomous intravenous insertion of aneedle. In an embodiment, the program 115 is configured to transformreal-time sensor data 92 acquired about a location of a patient's vesselinto a command to instruct one or more primary actuators 93 to executeautonomous intravenous insertion of a cannula.

FIG. 23 displays an embodiment diagram of the software architecture andflow of control of the main program designed to run the autonomousneedle insertion system 8. A veinbot structure 112 is created within themain program 115 to enable the user to interact with the system 8. Builtinto the veinbot structure 112 are a camera 61 and laser rangefinderunit 60, a robot connection and command module 113, and a GUI 120 whichenables the user to interact with the system 8 by selecting theprocedure to be done and monitoring it. Video may be processed indifferent ways. In an embodiment, the video is processed by a veinprocessor 112 a. In an embodiment, the video is processed by an armfinder 112 b. In an embodiment, the video is processed by an elbowfinder 112 c. Finding the patient's arm 112 b and elbow 112 c may beneeded for both suggesting best insertion sites and automatic targetsite selection, as they enable the vein selector 112 d to choose targetveins intelligently. In an embodiment, the vein selector 112 d onlychooses potential insertion sites inside the arm 112 b and near theelbow 112 c, which is where needles are typically inserted into theforearm by phlebotomists. The vein selector 112 d presents suitabletarget insertion sites either to the user (in manual mode) or to itself(in automatic mode), awaiting a target to be chosen. Once a target siteis chosen, a target vein structure 112 e can be created that contains animage of the chosen target. In an embodiment, the selected target istracked by the vein tracker 110 f of the vision system 110. The selectedtarget can be localized in 4 (x, y, z, u) or 6 (x, y, z, u, v, w)coordinates, depending on where in the insertion procedure the systemis. When the target is relatively far away, the localization unit 112 ftypically calculates 4 coordinates of the target site. Upon approachingthe target insertion site, the localization unit 112 f typicallycalculates 6 coordinates of the target insertion site for more preciseinsertion of the needle into the patient's arm 7 at the target insertionsite. In an embodiment, the localization unit 112 f calculates thecoordinates of the target insertion site by combining information fromthe target tracker 110 f of the vision system 110, the target veinstructure 112 e, and the laser rangefinder 60.

Following tracking and insertion site localization, control commandsfrom the robot connection and command module 113 may be sent to theslave control program 114, such as a command to move closer to thedesired target insertion site or a command for inserting the attachedmedical device into the insertion site. In an embodiment, the slavecontrol program 114 is a static program that awaits commands created bythe robot connection and command module 113. When commands are receivedfrom the robot connection and command module 113, the slave controlprogram 114 returns a signal to the master program 115 verifying thatthe correct command was received.

In an embodiment, the master program 115 comprises a control program forpermitting an operator of the system 8 to initiate an autonomousintravenous insertion procedure. In various embodiments, the masterprogram 115 is configured to analyze input data 92 acquired by the oneor more sensory devices 91, 213 and to transform the input data 92 intoa plurality of commands for output to the at least one actuator 93 toinstruct the actuator 93 to autonomously perform the medical procedure.In an embodiment, the program may be executed by a user in the vicinityof a patient receiving an intravenous insertion procedure. In anembodiment, the program may be executed remotely from a command center,as long as there exists communication between the master computer 90 andthe robot controller 1 a.

In an embodiment, the system 8 includes a control panel that enables auser of the system to choose a medical procedure to be performed. In anembodiment, the medical procedure comprises a needle insertion forphlebotomy. In an embodiment, the medical procedure comprises a cannulainsertion for phlebotomy. In an embodiment, the medical procedurecomprises a cannula insertion for drug administration. In an embodiment,the medical procedure uses one or more VACUTAINER® accessories toperform phlebotomy and drug administration.

It should be noted that, in an embodiment, a plurality of parameters forcontrolling the system 8 are set depending on the desired medicalprocedure to be performed. In an embodiment, the plurality of parametersinclude a parameter for needle insertion for phlebotomy. In anembodiment, the parameter for needle insertion for phlebotomy comprisesinstructions to load a needle from a needle loading site. In anembodiment, the parameter for needle insertion for phlebotomy comprisesinstructions to load a tube from a tube loading site. In an embodiment,the tube comprises a new tube. In an embodiment, the parameter forneedle insertion for phlebotomy comprises instructions to monitor howfull the loaded tubes are. In an embodiment, the parameter for needleinsertion for phlebotomy comprises instructions to change the tubes whennecessary or desirable. In an embodiment, the parameter for needleinsertion for phlebotomy comprises instructions to insert the needle. Inan embodiment, the parameter for needle insertion for phlebotomycomprises instructions to extract the needle. It should be appreciatedthat the instructions to insert and extract the needle can be carriedout according to well known needle insertion and extraction procedures.In an embodiment, the needles, tubes, and other equipment used compriseVACUTAINER® brand equipment.

In an embodiment, the plurality of parameters include a parameter forcannula insertion. In an embodiment, the parameter for cannula insertioncomprises an instruction to load the cannula from a cannula loadingsite. In an embodiment, the parameter for cannula insertion comprises aninstruction to extract a needle from a center of the cannula.

In an embodiment, the plurality of functions and methods executable bythe master computer 90 includes a get range function to instruct thelaser rangefinder 60 to obtain a distance, as described above. In anembodiment, the plurality of functions and methods executable by themaster computer 90 includes a get video function to instruct the NIRcamera 61 to capture a video of a patient's arm 7 in real-time.

In an embodiment, the plurality of functions and methods executable bythe master computer 90 includes a process video module or vein processor112 a to employ an image processing technique to transform the videocaptured of the patient's arm 7 from an image of the patient's arm 7 toan image of the patient's arm 7 showing useful information pertaining toa plurality of target insertion sites where a vessel of the patientexists. In an embodiment, the vein processor 112 a transforms thepreprocessed image of the patient's arm 7 acquired by the frame grabber110 a into a processed image of the patient's arm 7. Combined withinformation from the arm finder 112 b that discards visual informationnot inside the patient's arm 7 and the elbow finder 112 c that finds thepatient's elbow region, the processed image of the patient's arm 7 ispassed to the vein selector 112 d. In the vein selector 112 d, aplurality of bounding boxes may be projected onto the image of thepatient's arm 7, indicating target insertion sites for inserting amedical device into a vessel of the patient. In an embodiment, theprocessed image of the patient's arm 7 includes an image of the vesselof the patient featured in a first color which distinguishes the vesselfrom structures in the environment surrounding the vessel which arefeatured in a second, a third, a fourth, or any number of additionalcolors sufficient to show a contrast between the vessel and otherstructures in the environment surrounding the vessel.

In an embodiment, the plurality of functions and methods executable bythe master computer 90 includes using the vein selector 112 d to allowthe master computer 90 to autonomously select a target insertion sitefor automatically inserting a medical device tool 212 into a patient'svessel while the system 8 is operating in the automatic mode ofoperation.

In an embodiment, the plurality of functions and methods executable bythe master computer 90 includes using the vein tracker 110 f fortracking the target insertion site in real-time. In an embodiment, thevein tracker 110 f tracks the two-dimensional coordinates of the optimalinsertion site in real-time. In an embodiment, the vein tracker 110 ftracks the two-dimensional coordinates of the vessel in real-time. In anembodiment, the vein tracker 110 f tracks the orientation of the vesselin real-time.

In an embodiment, the plurality of functions and methods executable bythe master computer 90 build a target vein structure 112 e that holdsinformation about the targeted insertion site, including, but notlimited to, its appearance, size, shape, location, orientation, andtopography.

In an embodiment, the plurality of functions and methods executable bythe master computer 90 includes a vein localization unit 112 f forgenerating the three-dimensional coordinates of the patient's vessel andthe orientation of the patient's vessel. This functions by passing thecoordinates of the target vessel in the camera frame provided by thevein tracker 110 f with information from the rangefinder reader 110 cinto a function inside the localization unit 112 f with knowledge of howdistance to the target and pixels correspond to three-dimensional space.

In an embodiment, the plurality of functions and methods executable bythe master computer 90 includes a module to send commands and receiveinformation from the slave control program 114 for sending commands tothe at least one actuator 93 to autonomously perform a medical procedureand for receiving sensor data 92 acquired in real-time pertaining to theat least one actuator 93 so that the master computer 90 can process thesensor data 92 in real-time to make sure that the medical procedure issuccessful.

FIG. 24 is a flowchart of an embodiment operational procedure of theautonomous intravenous insertion system 8, beginning with a request fora certain procedure, and following through until the procedure has beensuccessfully carried out or aborted. FIG. 24 represents one insertionprocedure as a sequence of stages. The sequence begins with stage 140,where the system 8 waits to receive a request to perform an insertionprocedure on a patient. Once the patient is ready to have the procedureperformed, the system 8 is prompted to obtain medical device tools 212needed for the procedure in step 141 a. Simultaneously, the system 8applies the automatic tourniquet cuff 5 in step 141 b so that thepatient's forearm is stabilized and the vein within the forearm becomemore visible. Once the medical device tool 212 relevant to the procedurehas been obtained, the system 8 highlights potential insertion sitesand, depending on the mode of operation, either selects a target siteautomatically or receives input from the user regarding the targetinsertion site in step 142. With the site selected, the system tracksthe site in step 143, generates three-dimensional spatial coordinatesand an angle relative to the medical device tool 212 in step 144, andmoves continuously towards the insertion site until the medical devicetool 212 is directly above the insertion site in step 145. If anultrasound device 64 is included in the system 8, step 146 allows thesystem 8 to verify that a vein is indeed under the target insertionlocation, and in step 147 the system 8 determines the insertion depth.Once step 147 is complete, the system 8 inserts the butterfly needle 41,catheter 22, or other tool 80 in step 148. In step 149, the system 8proceeds to carry out procedure-specific tasks, such as engaging ordisengaging vacuum tubes in the case of drawing blood. When theprocedure is complete the system 8 discards the used butterfly needle41, catheter 22, or other tool 80 in step 150 and prepares itself tocarry out a new procedure in step 151.

In an embodiment, the request received from the operator to perform aninsertion procedure comprises an operator command to instruct theautonomous intravenous insertion system to perform a needle insertionfor phlebotomy. In an embodiment, the needle insertion for phlebotomycan be performed using VACUTAINER® blood drawing equipment. In anembodiment, the request received from the operator to perform theinsertion procedure comprises an operator instruction to operate theautonomous intravenous insertion system to perform a cannula insertionfor phlebotomy. In an embodiment, the request received from the operatorto perform an insertion procedure comprises an operator instruction tooperate the autonomous intravenous insertion system to perform a cannulainsertion for administration of a drug or for blood drawing. In anembodiment, the request received from the operator to perform aninsertion procedure comprises an operator instruction to operate theautonomous intravenous insertion system to perform a syringe insertionfor administration of a drug.

During step 140, the user may dictate the mode of selecting a target:automatic, semi-automatic, or manual. Automatic mode means the targetedvessel is selected entirely by the program, based on shape analysis andpre-programmed parameters to identify good insertion sites.Semi-automatic mode means the program will analyze the entire image andreturn in highly-distinguishable bounding boxes all sites it believesare apt for needle insertion. In this mode, the user must click withinthe box or on the vessel of interest on the video display 121 before thearm makes any movement. Manual mode refers to the user manuallyselecting the targeted vessel by clicking and dragging around thetargeted site. From there, the program will figure out the location ofthe targeted vessel, or if it determines that no vessel exists there, itwill prompt the user if he would like to pick a new target.

In an embodiment, once the request to perform an insertion procedure isreceived from the operator of the system at step 140, operation of theautonomous intravenous insertion system of the present disclosureproceeds to step 141 a and step 41 b. Step 141 a comprises obtaining amedical device tool 212 to be used to perform the insertion procedure.It should be appreciated that the medical device tool 212 obtained toperform the insertion procedure may depend on the particular insertionprocedure to be performed. In an embodiment, a plurality of medicaldevice tools 212 for performing the insertion procedure may be obtained.Step 141 b comprises activation of the system's automatic tourniquetcuff 5 and wrist stabilizing cuff 6, so as to increase the size andvisibility of the veins and demobilize the patient's arm.

Once the medical device tools 212 needed to perform the insertionprocedure are obtained, the tools 212 are then attached to an actuator93 of the autonomous intravenous insertion system. In an embodiment, themedical device tool 212 used to perform the insertion procedure ismanually attached by the operator of the system 8. In an embodiment, theoperator comprises a skilled technician. In an embodiment, the operatorcomprises an unskilled technician. In an embodiment, the medical devicetool 212 used to perform the insertion procedure is automaticallyattached by one of the actuators 93. In an embodiment, the medicaldevice tool 212 is automatically attached by the robot. The toolchangers 9 and 10 on the robot arm 1 change the tool 212 used to grabthe medical device based on the desired procedure. When the appropriatetool changer is in place, the medical device tools 212 needed to be usedfor the procedure are retrieved and fixed in place by the tool changergrabbing it. Both of these procedures may be done autonomously.

In an embodiment, the autonomous intravenous insertion system 8 of thepresent disclosure includes at least one loading station 11, 12 forloading and fastening the medical device tool 212 to be inserted to therobot arm 1. In an embodiment, the at least one loading station 11, 12includes a feeding mechanism for loading the medical device tool 212 tobe inserted to the robot arm 1. In an embodiment, the feeding mechanismis operable to load and fasten at least one ancillary device to therobot arm 1. In an embodiment, the loaded medical device may be grippedby the tool via pneumatic fasteners.

In an embodiment, the medical device tool 212 to be inserted comprises abutterfly needle 41. In an embodiment, the medical device tool 212 to beinserted comprises a syringe. In an embodiment, the medical device tool212 to be inserted comprises a cannula. In an embodiment, the medicaldevice tool 212 to be inserted comprises a blood drawing tube 78. In anembodiment, the blood drawing tube 78 comprises VACUTAINER® blooddrawing equipment. In an embodiment, the medical device tool 212 to beinserted comprises a catheter 22. In an embodiment, the catheter 22comprises BD Nexiva's Closed IV Catheter System. It should beappreciated that medical device tools 212 having similar functionalitiesto the medical device tools 212 disclosed herein may also be used.

In an embodiment, the at least one ancillary device used to perform theinsertion procedure is also obtained and subsequently positioned toperform the insertion procedure by the tool changer 9, 10. It should benoted that the ancillary devices used to perform the insertion proceduredepend on the insertion procedure to be performed.

Step 142 comprises, in an embodiment, identifying, using a mastercomputer 90 running the software described herein for identifying veins,a target insertion site for insertion of the medical device tool 212into a vessel of the patient's arm 7 during the insertion procedure. Toidentify the target insertion site for insertion of the medical devicetool 212, such as a butterfly needle 41 or catheter 22, into a vessel,the system 8 uses one or more primary sensors 91, such as an NIR camera61 or a laser rangefinder 60, to acquire real-time sensor data 92relating to the patient's vessels. In an embodiment, the system 8 mayalso use secondary sensors 213 such as a second camera if desired. In anembodiment, the autonomous intravenous insertion system 8 uses aninfrared imaging technique and a video processing technique, such as,for example, the ones described above, to automatically highlight aplurality of potential insertion locations on a video display 121 of thesystem 8 for the operator of the system to visualize. In an embodiment,in automatic mode, the system 8 automatically identifies the targetinsertion site for automatic insertion of the medical device tool 212into the vessel of the patient's arm 7. In an embodiment, insemi-automatic mode, the system 8 allows the operator of the system tomanually select the target insertion site from among the plurality ofpotential insertion locations highlighted on the display.

In an embodiment, the autonomous intravenous insertion system 8automatically highlights the plurality of potential insertion locationsfor the operator of the system to select for the insertion procedureusing a program configured to identify a vessel in the patient's arm 7.In an embodiment, the program operates in a semi-automatic mode in whichthe operator may select the target insertion site for insertion of themedical device tool 212 into the patient's vessel from among theplurality of insertion locations highlighted on the video display 121 byusing a mouse or other input device to click one of the plurality ofinsertion locations highlighted for the operator on the display. In anembodiment, the program can also be run in autonomous mode, where thetarget insertion site for insertion of the medical device tool 212 intothe patient's vessel is automatically and intelligently selected by thesystem 8 itself. In an embodiment, the program directs ampleillumination onto the patient's arm 7 and uses near-infrared imagingtechniques and contrast enhancement algorithm to obtain contrast of thepatient's vessel against the environment surrounding the vessel.

In an embodiment, once the target insertion site for insertion of themedical device tool 212 into the vessel of the patient's arm 7 isidentified and selected at step 142, either autonomously or via userinput, operation of the autonomous intravenous insertion system 8 of thepresent disclosure proceeds to step 143.

In an embodiment, step 143 comprises tracking, using the at least onesensor 93, such as an NIR camera 61, the target insertion site. In anembodiment, tracking the target insertion site occurs in real-time. Inan embodiment, automatically tracking the target insertion site inreal-time helps ensure that the actuators 93 guide the medical devicetool 212 to be inserted in the pre-determined location along an optimalinsertion path and an optimal orientation toward and into the patient'svessel.

A video camera 61 outfitted with a bandwidth filter permitting light inthe range of 720 nm to 780 nm to pass may be used to take pictures ofthe forearm. Such filter highlights the appearance of veins in the imagesince blood (hemoglobin) preferentially absorbs light in the NIR range.Thus, veins will appear comparatively darker through our camera thanthrough a standard white-light-sensing camera. An insertion site may beidentified and localized through the processing of image data 92received from the single NIR camera 61. The original input image isbinarized through a series of processing steps and then fed into atrained vein classifier. Before the image is fed into the classifier,some preprocessing is done to calculate a series of features having todo with each white blob in the processed image is calculated. Thesefeatures are related to the size, shape, thickness, length, and anglerelative to the entire arm of each blob. Based on these features, theclassifier makes a guess as to whether the blob is a vein or not.Together with the arm finder 112 b and elbow finder 112 c, a scoringmodule ranks the best veins, and presents them to the user in a highlydistinguishable manner. If the program is running in automatic mode, thevein with the best score is chosen as the target vein. Otherwise, theprogram waits until the user selects a suitable target.

Once a target has been selected, it is continuously tracked andultimately localized in three-dimensional space. Tracking requires notlosing the site within the image frame, and this is done with the videoprocessing system. This way, regardless of patient movement, the systemknows the location of the targeted site. Simultaneously, the robot arm 1can be commanded to move in the direction of the target site usingvisual feedback. The path taken leads the laser rangefinder 60 directlyover the insertion site, at which point the target location inthree-dimensional space can be determined. The topography of the aroundthe targeted site is also mapped using the laser rangefinder 60 so therobot arm 1 can be adjusted accordingly. The medical device tool 212 canthus be accurately positioned prior to inserting the tool 212 withoutaccidentally running into the patient's skin.

In an embodiment, the system 8 tracks the target insertion locationinstead of tracking the medical device tool 212 to be inserted for anumber of reasons. In an embodiment, the system 8 remains aware of theposition of the medical device tool 212 to be inserted throughout theinsertion procedure, and thus specifically tracking the medical devicetool 212 may be redundant. Since the system 8 is aware of the positionof the medical device tool 212, tracking the target insertion siterelative to the position of the medical device tool 212 to be insertedallows the system 8 to precisely position the medical device tool 212 inclose proximity to the target vessel at the target insertion site. Thisis desirable where a movement of the patient can cause the position ofthe medical device tool 212 to suddenly change relative to the targetinsertion site. Thus, tracking the target insertion site allows thesystem to update in real-time, using feedback acquired by the one ormore primary sensors 91, the position of the target insertion site, aswell as the optimal insertion path and orientation for the medicaldevice tool 212 to be inserted.

In an embodiment, the target insertion site for the medical device tool212 to be inserted into the patient's vessel is automatically tracked inreal time with respect to a movement of any kind of the patient. In anembodiment, the target insertion site is automatically tracked in realtime with respect to an arm movement of the patient. In an embodiment,the target insertion site is automatically tracked in real time withrespect to a voluntary arm movement of the patient. In an embodiment,the target insertion site is automatically tracked in real time withrespect to an involuntary arm movement of the patient. In an embodiment,the target insertion site is automatically tracked in real time withrespect to a movement of the patient's body that results in arepositioning of the patient's arm 7. In an embodiment, the targetinsertion site is automatically tracked in real time with respect torepositioning of the patient or the patient's arm 7. In an embodiment,the system may be programmed to abort the procedure, if an arm is nolonger seen within the field of view of the NIR camera 61.

In an embodiment, the target insertion site tracked in real-time by thesystem 8 comprises a perimeter on a skin surface of the patient in closeproximity to a vessel in the patient's arm 7 underneath the skinsurface. In an embodiment, the target insertion site is tracked inreal-time by the system 8 by projecting a bounding box onto an image ofthe patient's arm 7. In an embodiment, the bounding box encompasses aperimeter on the skin surface of the patient in close proximity to thevessel underneath the skin surface of the patient's arm 7.

In an embodiment, subsequent to or concomitantly with real-time trackingof the optimal insertion site at step 143, operation of the autonomousintravenous insertion system of the present disclosure proceeds to step144.

Step 144 comprises, in an embodiment, generating three-dimensionalcoordinates of the patient's vessel. This can be done by using themaster computer 90 to process the sensor 92 data acquired by the one ormore sensors 91, 213, such as the laser rangefinder 60 and at least oneNIR camera 61. In an embodiment, the three-dimensional coordinates ofthe targeted vessel are used by the master computer 90 to determine anoptimal insertion path and an optimal orientation for the medical devicetool 212 to be inserted into the patient's vessel. The system 8generates the three-dimensional coordinates of the optimal insertionpath and the optimal orientation continuously throughout the insertionprocedure. The system 8 continuously updates the three-dimensionalcoordinates of the target insertion site in real-time to prevent aspontaneous movement of the patient from making the insertion procedureunsuccessful.

In an embodiment, the system 8 is equipped with a laser measurementsystem precise to within 10 μm. The system 8 is calibrated to convertpixels to millimeters as a function of depth. A conversion function maybe used to determine physical distances from pixel dimensions in theimage. This conversion function may be based on the fixed geometry ofthe camera 61 and lens, and the variable distance of the lens to thetarget object is obtained by the laser measurement system. After thetarget insertion site is identified, the robot arm 1 may be commanded tomove the laser rangefinder 60 to a position several centimeters abovethe target insertion site using feedback from at least one imagingdevice as a guide. When the laser rangefinder 60 is in position andpointing at the target insertion site, the robot arm 1 may calculate thetopography of the insertion site with the laser rangefinder 60, so thatit may orient itself in such a way as to not be obstructed by thepatient's arm 7 during the insertion procedure. Next, the robot arm 1may orient itself in the appropriate fashion, and may be commanded tomove closer to the insertion target, using feedback from the imaginedevice and laser rangefinder 60 to place the medical device tool 212 inthe position where the insertion path will begin. After the targetinsertion site and orientation are identified, the robot arm 1 iscommanded to move the laser rangefinder 60 directly over the targetinsertion site using feedback from the at least one sensor 91, 213 as aguide. In an embodiment, the at least one sensor 91, 213 used to providefeedback to the system to command the robot arm 1 to move the laserrangefinder 60 over the target insertion site comprises an imagingdevice. In an embodiment, the imaging device is a camera 61. In anembodiment, the imaging device is an NIR camera 61. In an embodiment,the imaging device is an ultrasound imaging device 64. Once the depthneeded to reach the vessel underneath the skin surface of the targetinsertion site is found, its three-dimensional coordinates are generatedand transformed into a command to instruct the at least one actuator 93to move the medical device tool 212 to be inserted to this position.

In an embodiment, concomitantly with real-time tracking of the targetinsertion site at step 143 and the generation of the three-dimensionalcoordinates at step 144, operation of the autonomous intravenousinsertion system 8 of the present disclosure continues at step 145. Asillustrated in FIG. 24, step 145 comprises, in an embodiment,positioning, using the at least one actuator 93, the medical device tool212 in close proximity to the patient's vein, wherein when the tool 212is in close proximity to the patient's vein, the tool 212 is preferablypositioned at the beginning of a desired insertion path in the desiredorientation.

Due to patient's movement during the insertion procedure, a controlprogram on the maser computer 90 may be configured to evaluate thetarget insertion site in real-time so that when a movement of thepatient's arm 7 displaces the identified target insertion site withrespect to the position of the medical device tool 212 to be inserted,the control program sends in real-time updated three-dimensionalcoordinates of the target insertion site to a robot controller 1 a sothat the at least one actuator 93 can be repositioned for the medicaldevice tool 212 to be inserted through the target insertion site. Thatis, the robot arm 1 repositions itself in the optimal orientation so thetool 212 is in line with the targeted vessel. The medical device tool212 can thus be inserted through the target insertion site along anupdated optimal insertion path and an updated target insertion site sothat the medical device tool 212 is inserted safely and properly intothe patient's vessel.

Steps 143 through 145 may be carried out repeatedly in a loop until thesystem has determined it has reached the target insertion site.

In an embodiment, at least prior to step 147, operation of theautonomous intravenous insertion system of the present disclosureperforms step 146. Step 146 comprises, in an embodiment, verifying,using the at least one primary sensor 91, the existence of the veinunderneath the target insertion site. In an embodiment, verifying that avessel lies underneath the target insertion site ensures a safeinsertion procedure. In an embodiment, verifying that a vessel liesunderneath the target insertion site ensures a successful insertionprocedure. In an embodiment, verifying that a vessel lies underneath thetarget insertion site ensures an accurate insertion procedure. In anembodiment, verifying that a vessel lies underneath the target insertionsite reduces the risk of an improper insertion procedure. In anembodiment, verifying that a vessel lies underneath the target insertionsite reduces the risk of an error during the insertion procedure. In anembodiment, verifying that a vessel lies underneath the target insertionsite decreases the likelihood of having to repeat the insertionprocedure on the patient.

In an embodiment, at step 146 the autonomous intravenous insertionsystem 8 may deploy an ultrasound device 64 to verify the existence of avessel in the target location underneath the target insertion sitebeneath the patient's skin surface. In an embodiment, the system 8autonomously operates an ultrasound device 64 using the program on themaster computer 90 to instruct the at least one actuator 93 to performthe ultrasound imaging technique for precise positioning of theultrasound device 64 to obtain an ultrasound image of the patient'svessel underneath the target insertion site. In an embodiment,interpretation of the ultrasound image for verification that a vessel isin the specified location is performed by the ultrasound processor andvein verification system 65 of the system 8 of the present disclosureautonomously through.

The ultrasound image data may be processed using standard noisefiltering algorithms, and objects in the image are identified by theintensity of pixels comprising them. Specifically, veins will appear asdark, approximately circular objects in the image with radii of a fewmillimeters. Converting the images to binary images and excludingobjects of incorrect size and shape will highlight these veincross-sections, and allow the computer to determine their depth anddiameter.

In an embodiment, the user may view the ultrasound image generated bythe ultrasound device 64 and interpret the ultrasound image to verifythat the vessel is in the specified location. In an embodiment, theultrasound imaging device 64 is in a retracted position. In theretracted position, the ultrasound imaging device 64 does not obstructother sensors 91, 213 attached to the at least one actuator 93. In anembodiment, during the verification step, the ultrasound imaging device64 is moved from its retracted position into a position suitable bothfor verifying the existence of the vessel within the patient's arm 7underneath the skin surface of the target insertion site and formeasuring a depth of the vessel underneath the skin surface of thetarget insertion site. In an embodiment, when the ultrasound imagingdevice 64 is moved into the position suitable for verification andmeasurement of the vessel, other sensors 91, 213 attached to the atleast one actuator 93 may be obstructed from acquiring data 92 relatingto the patient's arm 7 and the vessel. However, it should be appreciatedthat at least one of the sensors 91, 213, such as the NIR camera 61,remains operable to track the patient's arm 7 in real-time and to detectmovement of the patient's arm 7. In an embodiment, the ultrasoundimaging device 64 includes an ultrasound probe. In an embodiment, anultrasound probe of the ultrasound imaging device 64 verifies theexistence of the vessel beneath the skin surface of the target insertionsite on the patient's arm 7. In an embodiment, the ultrasound device 64validates the real-time data 92 acquired by the at least one sensor 91in earlier steps.

Step 147 comprises, in an embodiment, determining, using either veinintensity, feedback from the ultrasound device 64, or preprogrammedinstructions, an insertion depth for the medical device tool 212 to beinserted into the patient's vessel along the optimal insertion path. Inan embodiment, the depth of the vessel underneath the target insertionsite is determined using an ultrasound signal or image obtained duringstep 146 through the ultrasound processor and vein verification system65 previously described. In an embodiment, the insertion depth ispreprogrammed but also monitored using feedback from the insertionprocedure, such as visual information indicating the vessel has beenpunctured, or force feedback along the axis of the medical device tool212. In the case where the insertion depth is preprogrammed, monitoringthe insertion depth with feedback is done to ensure the procedure issafe.

In an embodiment, at least prior to the step of inserting at step 148,the method for autonomous insertion of a medical device tool 212 into avessel of a patient to be treated comprises generating, using the mastercomputer 90, an updated optimal insertion path and an updated optimalorientation for the medical device tool 212 to be inserted based on amovement of the patient detected while using the at least one sensor 91to track the target insertion site in real-time.

In an embodiment, the master program 115 transforms in real-time theupdated desired insertion path and the desired orientation for themedical device tool 212 to be inserted into a command to instruct the atleast one actuator 93 to automatically reposition the medical devicetool 212 in the updated optimal insertion path and the optimalorientation to insert the medical device tool 212 into the patient'svessel.

In an embodiment, subsequent to the step of determining at step 147,operation of the autonomous intravenous insertion system 8 of thepresent disclosure proceeds to step 148.

As illustrated in FIG. 24, step 148 comprises, in an embodiment,inserting, using the at least one actuator 93, the medical device tool212 into the patient's vessel at the target insertion site. In anembodiment, the medical device tool 212 is inserted in the desiredorientation and along the desired insertion path. In an embodiment, themedical device tool 212 is inserted in a guided and steady manner intothe vessel of the patient. In an embodiment, the medical device tool 212is inserted in a straight line path into the vessel. In an embodiment,the medical device tool 212 is inserted in a straight line path into thevessel through the target insertion site to a depth equivalent to thedepth determined during step 147. In an embodiment, the vessel of thepatient comprises a vein. In an embodiment, the vessel of the patientcomprises a superficial vein. In an embodiment, the vessel of thepatient comprises a deep vein. In an embodiment, the vessel of thepatient comprises a subcutaneous vein. In an embodiment, the medicaldevice tool 212 comprises a butterfly needle 41. In an embodiment, themedical device comprises a cannula. In an embodiment, the medical devicetool 212 comprises a catheter 22.

In an embodiment, subsequent to the step of inserting at step 148,operation of the autonomous intravenous insertion system 8 of thepresent disclosure proceeds to step 149 which comprises performing oneor more tasks pertaining to a medical procedure to be performed inconnection with the insertion procedure to complete the medicalprocedure. In an embodiment, the autonomous intravenous insertion system8 of the present disclosure includes an auxiliary device connected tothe robot arm 1 for performing the one or more tasks pertaining to themedical procedure to be performed in connection with the insertionprocedure. In an embodiment, the one or more tasks to be performed bythe auxiliary device comprises engaging a blood drawing tube 78. In anembodiment, the one or more tasks to be performed by the auxiliarydevice comprises disengaging a blood drawing tube 78. In an embodiment,the one or more tasks to be performed by the auxiliary device comprisesswapping a first blood drawing tube 78 with one or more blood drawingtubes 78. In an embodiment, the blood drawing tube 78 comprises aVACUTAINER® tube.

In an embodiment, subsequent to at least the step of inserting at step148, operation of the autonomous intravenous insertion system 8 of thepresent disclosure continues to step 150 which comprises automaticallywithdrawing the medical device tool 212 for subsequent disposal of themedical device tool 212 for sanitary purposes. In an embodiment,withdrawal of the medical device tool 212 comprises extracting the tool212 by retracing the tool 212 outward along the optimal insertion pathused to insert the medical device tool 212 into the vessel.

In an embodiment, once the medical device tool 212 is withdrawn anddiscarded into the disposal unit 96 at step 150, operation of theautonomous intravenous insertion system 8 proceeds to step 151. At step151, the autonomous intravenous insertion system 8 returns to step 140and waits to receive instructions from an operator of the system 8 torequest an insertion procedure to be performed on a patient.

FIG. 25, FIG. 26 and FIG. 27 show an embodiment of the GUI 120 for theautonomous intravenous insertion system 8 of the present disclosure. Inan embodiment, the GUI 120 is displayed on a touch screen connected tothe system 8. However, the GUI 120 may be presented on any type ofscreen as long as the user has means to interact with it. FIG. 25illustrates an embodiment of the home screen. Referring to FIG. 25, thevideo input from the NIR camera 61 is displayed in the video display121. The user can select from one of two processors, original processor122 a or full processor 122 b. In an embodiment, the original processor122 a looks for short, stubby vein-like blobs in the processed image.The full processor 122 b selects veins using a machine learningalgorithm which has been trained to identify forearm veins inconjunction with a module that keeps track of veins found over time. Inan embodiment, keeping track of veins over time may help to eliminatesthe chance of false-positives from being selected. The user may alsoselect how the user intends to find a target insertion site. “Manualselection” 123 a denotes that the user will select the site himself fromclicking and dragging on the target site or clicking on a suggestedtarget. “Automatic selection” 123 b will select the target insertionsite automatically for the user. Buttons 124 and 125 allow the user toswitch between viewing modes, raw image 124 or processed image 125. Thestatus of the robot arm 1 is also displayed to the user at robot statusindicator box 126. The main sensor tool connected option 127 indicatesthat the main sensor tool containing the NIR camera 61 and the laserrangefinder 60 on the robot arm 1 are connected and working. The robotconnected option 128 indicates that the robot motor controller is on andthe robot is ready for actuation. In an embodiment, for insertionprocedures, both “Main Tool Connected” 127 and “Robot Connected” 128 aremarked, and “Demo Stand Connected” 129 is not marked. On the other hand,“Demo Stand Connected” 129 is marked when simply testing the system'svein finding, as shown in the screen shot in FIG. 25. From the homescreen, the user can also select from a number of actions 130 the robotarm 1 can perform. In an embodiment, options are “Draw Blood” 130 aindicating the system will carry out a blood drawing procedure, “InsertIV” 130 b indicating the system will carry out an IV insertionprocedure, “Dry Run” 130 c indicating the user wants to test the visionsystem 110 vein selection and tracking capabilities while the robotapproaches the target site, “Vision Demo” 130 d indicates the user wantsto demonstrate or test the vein processor's capabilities, and“Properties” 130 e allows the user to change vision processingproperties.

In an embodiment, upon selecting “Vision Demo” 130 d from the actions130, the user is presented with the screen as shown in FIG. 26. In thescreen shot of FIG. 26, the processor mode is “Full Processor” 122 b andthe selection mode is on “Manual Selection” 123 b. As shown, the system8 suggests potential target insertion sites to the user by placingbounding boxes 135 around the sites. To select a target insertion site,the user may either click the bounding box with a target vein inside itor click and drag a box over the desired target in either of the videodisplays shown 121 a, 122 b. Once a target insertion site is selected,it is tracked in real time. To reset the target insertion site, the usercan use the “Reset” button 134 a in one of the possible actions 134,shown on the right in FIG. 26. In an embodiment, other actions 134include the “Properties” button 134 b, which enables the user to alterthe processing properties, and the “Home” button 134 c which takes theuser to the home screen.

The video display 121 displays a processed vein image. In an embodiment,the system 8 provides such processed image to the vein identificationsystem 110 d, discussed above. Options “Show Processed Veins” 132 and“Show Arm Mask” 133 enable the user to see either the processed veinimage (shown above) or the “arm mask” output by the arm finder(reference 112 b not shown here) which outlines the patient's arm 7 inreal time. Atop the image lies a “Display Options” list 131 that enablesthe user to see various objects found by the vein identification system.

FIG. 27 shows an embodiment screen shown in response to selectingoptions from the Display Options list 131. Suitable veins are outlinedby providing bounding boxes outlining target vein suggestions 135 aboutthem. In addition, the elbow center estimate is displayed by a dot 136,which gives the user an idea of the point around which to look fortarget veins. More formally, the “Show Elbow Region” option displays aborder around the region in which to look for target sites by drawing aboundary 137 around the region. Options “Box Current” and “Show Current”enable the user to bypass the software module that keeps track of veinsfound over time and displays every vein as estimated by the learned veinmodel.

The systems and methods disclosed herein are controlled by a computerconfigured to acquire using at least one sensory device real-time datarelating to a patient and to transform the real-time data intoinformation useful to select an target insertion site for a target vein,and to transform the real-time data into a command output to at leastone actuator to instruct the actuator to autonomously perform aninsertion procedure wherein a medical device, such as a needle or acannula is inserted into the target vein at the target insertion site.In an embodiment, the computer is a master computer configured tocommunicate with a robot arm at least via an Ethernet port. The mastercomputer running the control program can be thought of as a stand-aloneunit that can communicate with the robot arm in myriad ways including,but not limited to, Ethernet communication, serial communication, orcommunication over a wireless network.

FIG. 28 is a block diagram illustrating an internal architecture of anexample of a computer, such as the master computer 90, in accordancewith one or more embodiments of the present disclosure. A computer asreferred to herein refers to any device with a processor capable ofexecuting logic or coded instructions, and could be a server, personalcomputer, set top box, smart phone, pad computer or media device, toname a few such devices. As shown in the example of FIG. 28, internalarchitecture 220 includes one or more processing units (also referred toherein as CPUs) 226, which interface with at least one computer bus 221.Also interfacing with computer bus 221 are persistent storagemedium/media 223, network interface 227, memory 222, e.g., random accessmemory (RAM), run-time transient memory, read only memory (ROM), etc.,media disk drive interface 224 as an interface for a drive that can readand/or write to media including removable media such as floppy, CD-ROM,DVD, etc. media, display interface 225 as interface for a monitor orother display device, keyboard interface 228 as interface for akeyboard, pointing device interface 229 as an interface for a mouse orother pointing device, and miscellaneous other interfaces not shownindividually, such as parallel and serial port interfaces, a universalserial bus (USB) interface, and the like.

Memory 222 interfaces with computer bus 221 so as to provide informationstored in memory 222 to CPU 226 during execution of software programssuch as an operating system, application programs, device drivers, andsoftware modules that comprise program code, and/or computer-executableprocess steps, incorporating functionality described herein, e.g., oneor more of process flows described herein. CPU 226 first loadscomputer-executable process steps from storage, e.g., memory 222,storage medium/media 223, removable media drive, and/or other storagedevice. CPU 226 can then execute the stored process steps in order toexecute the loaded computer-executable process steps. Stored data, e.g.,data stored by a storage device, can be accessed by CPU 226 during theexecution of computer-executable process steps.

Persistent storage medium/media 223 is a computer readable storagemedium(s) that can be used to store software and data, e.g., anoperating system and one or more application programs. Persistentstorage medium/media 223 can also be used to store device drivers, suchas one or more of a digital camera driver, monitor driver, printerdriver, scanner driver, or other device drivers, web pages, contentfiles, playlists and other files. Persistent storage medium/media 223can further include program modules and data files used to implement oneor more embodiments of the present disclosure.

For the purposes of this disclosure a computer readable medium storescomputer data, which data can include computer program code that isexecutable by a computer, in machine readable form. By way of example,and not limitation, a computer readable medium may comprise computerreadable storage media, for tangible or fixed storage of data, orcommunication media for transient interpretation of code-containingsignals. Computer readable storage media, as used herein, refers tophysical or tangible storage (as opposed to signals) and includeswithout limitation volatile and non-volatile, removable andnon-removable media implemented in any method or technology for thetangible storage of information such as computer-readable instructions,data structures, program modules or other data. Computer readablestorage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM,flash memory or other solid state memory technology, CD-ROM, DVD, orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other physical ormaterial medium which can be used to tangibly store the desiredinformation or data or instructions and which can be accessed by acomputer or processor.

For the purposes of this disclosure a module is a software, hardware, orfirmware (or combinations thereof) system, process or functionality, orcomponent thereof, that performs or facilitates the processes, features,and/or functions described herein (with or without human interaction oraugmentation). A module can include sub-modules. Software components ofa module may be stored on a computer readable medium. Modules may beintegral to one or more servers, or be loaded and executed by one ormore servers. One or more modules may be grouped into an engine or anapplication.

In an embodiment, the autonomous intravenous insertion system of thepresent disclosure includes a sensor for acquiring data in real-time,wherein the data relates to a vessel of a patient; an actuator forautomatic completion of a medical procedure, and a computer configuredto transform the data acquired by the sensor into a command to instructthe actuator to automatically complete the medical procedure.

In an embodiment, the autonomous intravenous insertion system of thepresent disclosure includes at least one sensory device for acquiringdata in real-time, wherein the data relates to three-dimensionalcoordinates of a vessel of a patient receiving a medical procedure; atleast one actuator for autonomous completion of the medical procedure;and a computer configured to transform the data from the coordinates ofthe vessel into a command to instruct the actuator to automaticallycomplete the medical procedure.

In an embodiment, the autonomous intravenous insertion system of thepresent disclosure includes a plurality of sensory devices for acquiringdata in real-time, wherein the data relates to three-dimensionalcoordinates of a vessel of a patient receiving a medical procedure; aplurality of actuators for autonomous completion of the medicalprocedure; and a computer configured to transform the data in real-timefrom the three-dimensional coordinates of the vessel into a command toinstruct the actuators to automatically complete the medical procedure.

In an embodiment, the autonomous intravenous insertion system of thepresent disclosure includes a sensory device for acquiring data inreal-time, wherein the data indicates three-dimensional coordinates of avein of a patient receiving an insertion procedure; an actuator forautonomous completion of the insertion procedure; and a computerconfigured to transform the data in real-time from the three-dimensionalcoordinates of the vein into a command to instruct the actuators toautomatically insert a medical device into the vein of the patient tocomplete the insertion procedure. In an embodiment, the medical devicecomprises a needle. In an embodiment, the medical device comprises acannula.

In an embodiment, the autonomous intravenous insertion system of thepresent disclosure includes one or more sensory devices for acquiringthree-dimensional coordinates and orientation of a patient's vein; oneor more actuators for autonomous insertion of a needle or cannula intothe patient's vein; and a computer configured to transform the acquiredthree-dimensional coordinates and orientation of the patient's vein intoan optimal insertion path for the needle or cannula to be inserted intothe patient's vein, wherein the sensory devices track the optimalinsertion path for the needle or cannula in real-time so that thecomputer can execute a command to the actuators to instruct theactuators to guide the needle or cannula along the optimal insertionpath to insert the needle or cannula into the patient's vein.

In an embodiment, a method for autonomous intravenous insertion of amedical device into a vessel of a patient comprises providing anautonomous intravenous insertion system comprising at least one sensorydevice for acquiring data relating to the patient's vessel, at least oneactuator for autonomous insertion of the medical device into thepatient's vessel, and a computer configured to transform the acquireddata into a command to instruct the at least one actuator toautomatically insert the medical device into the patient's vessel;acquiring, using the at least one sensory device, real-time datarelating to the patient's vessel, generating, using the computer, anoptimal insertion path for the medical device to be inserted into thepatient's vessel, wherein the optimal insertion path is generated basedon real-time data acquired by the sensory device; transforming, usingthe computer, the three-dimensional coordinates and orientation of thepatient's vessel into a command to instruct the at least one actuator toautomatically insert the medical device into the patient's vessel; andinserting, using the at least one actuator, the medical device into thepatient's vessel along the optimal insertion path generated by thecomputer.

In an embodiment, a method for autonomous intravenous insertion of aneedle into a vein of a patient comprises providing an autonomousintravenous insertion system comprising at least one sensory device foracquiring data relating to the patient's vein, at least one actuator forautonomous insertion of the needle into the patient's vein, and acomputer configured to transform the data acquired by the sensory deviceinto a command to instruct the at least one actuator to automaticallyinsert the needle into the patient's vein; acquiring, using the at leastone sensory device, real-time data relating to the patient's vein,wherein the real-time data comprises three-dimensional coordinates ofthe patient's vein; generating, using the computer, an optimal insertionpath for the needle to be inserted into the patient's vein;transforming, using the computer, the three-dimensional coordinates ofthe patient's vein into a command to instruct the at least one actuatorto automatically insert the needle into the patient's vein; andinserting, using the at least one actuator, the needle into thepatient's vein along the optimal insertion path.

In an embodiment, a method for autonomous intravenous insertion of acannula into a vein of a patient comprises providing an autonomousintravenous insertion system comprising at least one sensory device foracquiring data relating to the patient's vein, at least one actuator forautonomous insertion of the cannula into the patient's vein, and acomputer configured to transform the data acquired by the sensory deviceinto a command to instruct the at least one actuator to automaticallyinsert the cannula into the patient's vein; acquiring, using the atleast one sensory device, real-time data relating to the patient's vein,wherein the real-time data comprises three-dimensional coordinates ofthe patient's vein; generating, using the computer, an optimal insertionpath for the cannula to be inserted into the patient's vein;transforming, using the computer, the three-dimensional coordinates ofthe patient's vein into a command to instruct the at least one actuatorto automatically insert the cannula into the patient's vein; andinserting, using the at least one actuator, the needle into thepatient's vein along the optimal insertion path.

In an embodiment, a method for autonomous intravenous insertion of aneedle into a vein of a patient comprises providing an autonomousintravenous insertion system comprising at least one sensory device foracquiring data relating to the patient's vein, at least one actuator forautonomous insertion of the needle into the patient's vein, and acomputer configured to transform the data acquired by the sensory deviceinto a command to instruct the at least one actuator to automaticallyinsert the needle into the patient's vein; acquiring, using the at leastone sensory device, real-time data relating to the patient's vein,identifying, using the computer, an target insertion site for the needleto be inserted into the patient's vein; tracking, using the at least onesensory device, the target insertion site, wherein tracking the targetinsertion site occurs in real-time; generating, using the computer toprocess the data acquired by the sensory devices, three-dimensionalcoordinates of the patient's vein, wherein the three-dimensionalcoordinates are used by the computer to determine an optimal insertionpath and an optimal orientation for the needle to be inserted into thepatients vein; positioning, using the at least one actuator, the needlein close proximity to the patient's vein, wherein when the needle is inclose proximity to the patient's vein, the needle is positioned in theoptimal insertion path and the optimal orientation; verifying, using theat least one sensory device, the existence of the vein underneath thetarget insertion site; determining, using the at least one sensorydevice, an insertion depth for the needle to be inserted into the veinalong the optimal insertion path; generating, using the computer, anupdated optimal insertion path and an updated optimal orientation forthe needle to be inserted based on a movement of the patient detectedwhile using the at least one sensory device to track the targetinsertion site in real-time; transforming, using the computer, theupdated optimal insertion path and the optimal orientation for theneedle to be inserted in real-time into a command to instruct the atleast one actuator to automatically reposition the needle in the updatedoptimal insertion path and the optimal orientation to insert the needleinto the patient's vein; and inserting, using the at least one actuator,the needle into the patient's vein along the optimal insertion path.

In an embodiment, there is provided an autonomous intravenous needleinsertion system that may be comprised of a robotic arm for positioningthe needle, a robot controller for converting commands into movement, amaster computer for interpreting data and sending commands to the robotand other actuators, sensors for locating veins including but notlimited to a NIR camera and Laser Rangefinder, quick-change tooling formanipulation of different medical devices with the same system, andvarious support devices such as dispensers for new needles and bloodcollection tubes, storage for blood samples, a disposal unit, anautomated tourniquet, an ambient lighting control unit, and a dockingstation for various manipulators.

In an embodiment, there is provided a control program that incorporatesdata from sensors such as cameras, rangefinders and ultrasound probes,converts that data into information about the 3-dimensional location,orientation and local topography of veins in a human body, and uses thedata to command a robot arm to insert the needle safely into the targetvein, using tracking algorithms to keep targets in sight in case ofpatient movement.

In an embodiment, there is provided a control program for completing anentire intravenous needle insertion procedure autonomously, includingthe loading and manipulating of devices commonly associated withintravenous insertion procedures, gathering of data about the subject,inserting needles into veins reliably and accurately, and discardingused equipment.

In an embodiment, there are provided control functions to handleautonomous needle insertion of everyday medical devices on humans, suchas syringes, VACUTAINER® tubes, cannulas, and catheters

In an embodiment, there are provided control functions to control andhandle all tasks and procedures associated with needle insertionprocedures

In an embodiment, there is provided a method of finding superficialveins using ambient light (natural sunlight or other sources thatproduce diffuse near-infrared light) and a video camera with anear-infrared (NIR) filter to block all but wavelengths that areabsorbed by blood significantly more than by skin, water or surroundingtissues, and processing the resulting image into a binary image whereone color represents veins and the other represents everything else.

In an embodiment, there is provided a method of analyzing binary imagesof veins to determine defining characteristics such as length, diameter,orientation and position relative to other veins, and using thatinformation to determine the best targets for intravenous needleinsertion based on medical science, patient history and history ofsuccessful insertions performed by the robot on all subjects.

In an embodiment, there is provided an algorithm for scoring potentialinsertion sites based on length, diameter, orientation and positionrelative to the subject's elbow.

In an embodiment, there is provided a method of moving a camera andrangefinder with a robot arm to determine 3D coordinates of an object inthe robot's coordinate system using geometry of the assembly and thecamera lens

In an embodiment, there is provided a method for calibrating the abovesystem without human input, whereby the robot moves the assemblygathering data about a simple object such as a black circle on a pieceof paper, and uses the changes in shape and size resulting fromdifferent movements to determine the conversion factor between pixels inan image and physical dimensions in space for any given reading from therangefinder.

In an embodiment, there is provided a method for tracking an object witha robot arm in real time using the above system to quickly determinerelative distances between the robot arm and the target, and using thatinformation to follow the target.

In an embodiment, autonomous intravenous insertion systems and methodsfor using the system for autonomous insertion of a needle or cannulainto a vessel are disclosed herein. In an embodiment, there is providedan autonomous intravenous insertion system that includes one or moresensory devices for acquiring three-dimensional coordinates andorientation of a patient's vessel; one or more actuators for autonomousinsertion of a needle or cannula into the patient's vessel; and acomputer configured to transform the acquired three-dimensionalcoordinates and orientation of the patient's vessel into an optimalinsertion path for the needle or cannula to be inserted into thepatient's vessel, wherein the sensory devices track the optimalinsertion site in real-time so that the computer can execute a commandto the actuators to instruct the actuators to guide the needle orcannula to that site along a suitable insertion path to insert theneedle or cannula into the patient's vessel.

In an embodiment, there is provided a method for autonomous intravenousinsertion of a needle into a vessel of a patient that includes providingan autonomous intravenous insertion system comprising at least onesensory device for acquiring data relating to the patient's vessel, atleast one actuator for autonomous insertion of the needle into thepatient's vessel, and a computer configured to transform the dataacquired by the sensory device into a command to instruct the at leastone actuator to automatically insert the needle into the patient'svessel; acquiring, using the at least one sensory device, real-time datarelating to the patient's vessel, wherein the real-time data comprisesthree-dimensional coordinates of the patient's vessel; generating, usingthe computer, an optimal insertion path for the needle to be insertedinto the patient's vessel; transforming, using the computer, thethree-dimensional coordinates of the patient's vessel into a command toinstruct the at least one actuator to automatically insert the needleinto the patient's vessel; and inserting, using the at least oneactuator, the needle into the patient's vessel along the optimalinsertion path.

In an embodiment, there is provided a method of using an intravenousinsertion system that includes providing an autonomous intravenousinsertion system comprising at least one sensory device for acquiringdata relating to the patient's vessel, at least one actuator forautonomous insertion of the cannula into the patient's vessel, and acomputer configured to transform the data acquired by the sensory deviceinto a command to instruct the at least one actuator to automaticallyinsert the cannula into the patient's vessel; acquiring, using the atleast one sensory device, real-time data relating to the patient'svessel, wherein the real-time data comprises three-dimensionalcoordinates of the patient's vessel; generating, using the computer, anoptimal insertion path for the cannula to be inserted into the patient'svessel; transforming, using the computer, the three-dimensionalcoordinates of the patient's vessel into a command to instruct the atleast one actuator to automatically insert the cannula into thepatient's vessel; and inserting, using the at least one actuator, theneedle into the patient's vessel along the optimal insertion path.

In an embodiment, there is provided a method of using an intravenousinsertion system that includes providing an autonomous intravenousinsertion system comprising at least one sensory device for acquiringdata relating to the patient's vessel, at least one actuator forautonomous insertion of a VACUTAINER® needle into the patient's vessel,and a computer configured to transform the data acquired by the sensorydevice into a command to instruct the at least one actuator toautomatically insert the cannula into the patient's vessel; acquiring,using the at least one sensory device, real-time data relating to thepatient's vessel, wherein the real-time data comprises three-dimensionalcoordinates of the patient's vessel; generating, using the computer, anoptimal insertion path for the cannula to be inserted into the patient'svessel; transforming, using the computer, the three-dimensionalcoordinates of the patient's vessel into a command to instruct the atleast one actuator to automatically insert the cannula into thepatient's vessel; inserting, using the at least one actuator, the needleinto the patient's vessel along the optimal insertion path; and drawingone or more blood samples, using at least one actuator, by puncturingempty VACUTAINER® tubes according to procedure specifications.

In an embodiment, the robot arm described is used merely as a needlepositioning device. In an embodiment, other configurations of actuatorscan be used to position the tools used to manipulate the contemplatedmedical devices, In an embodiment, other systems of actuators may differfrom the robot arm of the present disclosure in that they may notconsist entirely of revolute joints. In an embodiment, an actuator mayinclude a combination of 3 or more prismatic (linear) joints to positionthe needle in three dimensions (x, y, and z) and 2 or more revolutejoints to align the needle with the target vessel and arm contour may beused.

In an embodiment, the tool used to manipulate the medical device may ormay not have an actuator on board to insert the needle. In anembodiment, in the case of the Catheter Tool, the robot arm inserts theneedle, whereas in the case of the Blood Drawing Tool, the tool itselfhas an on-board actuator that takes care of inserting and extracting theneedle. In an embodiment, a tool used for inserting catheters includesan on-board actuator for inserting the catheter into the target vessel.In an embodiment, the robot arm or other system of actuators to insertblood drawing needles may be used in place of the Blood Drawing Tool'son-board actuator. In an embodiment, any type of automated or usertriggered mechanical needle insertion mechanism may be used as part ofsystem 8 for inserting a needle into the subject arm.

In an embodiment, other working mechanisms may be used to achieve thesame outcome as the mechanisms embedded in the tools described herein.

In an embodiment, it should be noted that the needle to be inserted isvisible to the patient. In an embodiment, the needle might be hiddenfrom a patient's view to limit discomfort elicited from viewing needles.In one embodiment, insertable devices may be hidden in the tool thatmanipulates them until the time when the needle is about to be inserted.In another embodiment, the needle is completely hidden from the patientthroughout the entire procedure. In this embodiment, the needle may bekept within the body of a tool.

In an embodiment, the sensors are mounted onto the robot arm, and visualserving techniques are used to position the needle accurately. Othersensor configurations may exist. For instance, in an embodiment, thesensors are in a fixed position above target insertion area. In anembodiment, the sensors are mounted on their own positioning systemabove the target insertion area, and are autonomously positioned to bedirectly over the target insertion site.

In an embodiment, no ultrasound imaging unit is appended. In anembodiment, the ultrasound imaging device is manipulated by its ownmanipulator and appended to the robot arm. In an embodiment, theultrasound imaging device has its own manipulator stationed nearby thetarget insertion site, and is autonomously deployed before and while theinsertion procedure is underway. In an embodiment, the ultrasound probeis built in to the tools used to manipulate medical devices. In thisembodiment, the tool may make contact with the patient, and therefore,the needle may be hidden within the tool to avoid premature contact withthe patient.

In an embodiment, the sensors used to localize a target vessel are a NIRonly camera and a laser rangefinder. In an embodiment, other threedimensional localization systems are used. Other three dimensionallocalization systems include, but are not limited to, stereo cameras, asystem of two or more cameras (not necessarily aligned as stereo camerasare), LADAR, LIDAR or other similar range finding unit, a system of oneor more cameras combined with one or more laser rangefinders, andultrasound.

In an embodiment, force feedback methods for sensing whether or not aneedle has penetrated the skin, whether or not a needle has penetrated avessel and whether or not a vessel is present in the target location areused. In an embodiment, a force sensor may be utilized in conjunctionwith a positioning device to obtain such information.

In an embodiment, a system for autonomous intravenous insertion includesa robot arm, one or more sensors pivotally attached to the robot arm forgathering information about potential insertion sites in a subject arm,a medical device pivotally attached to the robot arm, and a controllerin communication with the sensors and the robot arm, wherein thecontroller receives the information from the sensors about potentialinsertion sites, and the controller selects a target insertion site anddirects the robot arm to insert the medical device into the targetinsertion site.

In an embodiment, a system for autonomous intravenous insertion includesa robot arm, a plurality of sensors attached to the robot arm forgathering information about potential insertion sites in a subject arm,a medical device holding tool detachably engaged to the robot arm, thetool comprising a plurality of grippers for holding a medical device tobe inserted into the subject arm, a first actuating mechanism foractuating the plurality grippers, stabilizing feet; and a secondactuating mechanism for placing the stabilizing feet in the proximity toan insertion site, and a controller in communication with the pluralitysensors, the medical device holding tool, and the robot arm, wherein thecontroller receives the information from the sensors about potentialinsertion sites, and selects a target insertion site and directs themedical device holding tool and the robot arm to insert the medicaldevice into the target insertion site.

In an embodiment, a method for autonomous intravenous insertion includessecuring a subject arm, identifying a target insertion site based oninformation received from at least one sensor, actuating a robot arm todeliver a medical device to the target insertion site, while monitoringthe target insertion site, and inserting the medical device into thesubject arm at the insertion site.

All patents, patent applications, and published references cited hereinare hereby incorporated by reference in their entirety. It will beappreciated that several of the above-disclosed and other features andfunctions, or alternatives thereof, may be desirably combined into manyother different systems or applications. Various presently unforeseen orunanticipated alternatives, modifications, variations, or improvementstherein may be subsequently made by those skilled in the art which arealso intended to be encompassed by the following claims.

What we claim is:
 1. A method for autonomous intravenous needleinsertion comprising: receiving, from one or more sensors, informationabout potential insertion sites in blood vessels of the subject arm;identifying a target insertion site based on the information receivedfrom the one or more sensors; and autonomously directing a needle to thetarget insertion site and inserting the medical device into the subjectarm at the target insertion site.
 2. The method of claim 1 wherein thestep of directing further comprises monitoring the target insertion siteand tracking the insertion site through movements of the subject.
 3. Themethod of claim 1 wherein the step of identifying comprises: receivingan image of the subject arm from a camera; identifying potentialinsertion sites; ranking potential insertion sites based on size, shape,and location of the potential insertion sites; and prioritizingpotential insertion sites based on the ranking.
 4. The method of claim 1further comprising selecting the target insertion site based on aranking of the potential insertion sites.
 5. The method of claim 1wherein the step of identifying comprises: highlighting and presentingto the user potential insertion sites; and enabling the user to selectthe target insertion site.
 6. The method of claim 1 wherein the step ofidentifying comprises: determining the location of an elbow in thesubject arm; ranking potential targets depending on the locationrelative to the elbow; and selecting the target insertion site based onthe ranking of the potential insertion sites.
 7. A method of claim 1further comprising automatically stabilizing the vessel prior to theinserting of the needle into the target site.
 8. The method of claim 1further comprising using an ultrasound sensor to verify the existence ofa blood vessel at the target insertion site and determining a depth ofthe blood vessel from a surface of the subject arm.
 9. The method ofclaim 1 further comprising receiving information from a rangefinder toorient the needle for insertion at the target insertion site.
 10. Themethod of claim 1 further comprising identifying the target insertionsite in at least 4 (x, y, z, u) coordinates to guide the insertion ofthe needle into the subject arm at the target insertion site.
 11. Themethod of claim 1 wherein the information about potential insertionsites comprises topography of the subject arm.
 12. A method forautonomous intravenous needle insertion comprising: processinginformation from one or more sensors to locate potential insertion sitesin blood vessels of a subject; autonomously selecting a target insertionsite in a target blood vessel of the subject into which to insert aneedle; autonomously calculating an insertion path and an insertionangle of the needle into the target blood vessel based on theinformation from the one or more sensors; and autonomously directing theneedle along the insertion path and at the insertion angle into thetarget blood vessel at the target insertion site.
 13. The method ofclaim 12 wherein the information comprises information about topographyof the arm of the subject.
 14. The method of claim 13 wherein theinformation further includes three-dimensional coordinates andorientation of the target vessel and an orientation and a position ofthe needle.
 15. The method of claim 14 further comprising autonomouslytracking the position and orientation of the target insertion site inreal-time during the directing of the needle.
 16. The method of claim 12wherein the one or more sensors comprises an ultrasound probe.
 17. Themethod of claim 12 further comprising gathering information aboutpotential insertion sites in blood vessels of a subject with anintravenous needle insertion system, the intravenous needle insertionsystem comprising: a robotic positioning system; an insertion moduleremovably attachable to the robotic positioning system, the insertionmodule comprising: a sensor assembly detachably engaged to the insertionmodule, the sensor assembly including one or more sensors attached tothe sensor assembly to gather the information about potential insertionsites in blood vessels of the subject; and a needle holding tooldetachably engaged to the insertion module, the needle holding toolcomprising a needle to be inserted into the subject.
 18. The method ofclaim 17 wherein the needle holding tool further comprises a pluralityof grippers for holding the needle; a first actuating mechanism toactuate the plurality of grippers; stabilizing feet; a second actuatingmechanism to place the stabilizing feet in the proximity to an insertionsite; and a third actuating mechanism to advance the needle into atarget blood vessel at a target insertion site.
 19. An intravenousneedle insertion system comprising: a robotic positioning system; aninsertion module removably attachable to the robotic positioning system,the insertion module comprising: a sensor assembly detachably engaged tothe insertion module, the sensor assembly including one or more sensorsattached to the sensor assembly to gather information about potentialinsertion sites in blood vessels of a subject; and a needle holding tooldetachably engaged to the insertion module, the needle holding toolcomprising a needle to be inserted into the subject; and a controller incommunication with the one or more sensors, the needle holding tool, andthe robotic positioning system, wherein the controller is configured toautonomously process the information from the plurality of sensors toselect the target insertion site in the target blood vessel of thesubject into which to insert the needle, and to autonomously direct theneedle into the target blood vessel at the target insertion site. 20.The system of claim 19 wherein the needle holding tool furthercomprises: a plurality of grippers for holding the needle; a firstactuating mechanism to actuate the plurality of grippers; stabilizingfeet; a second actuating mechanism to place the stabilizing feet in theproximity to an insertion site; and a third actuating mechanism toadvance the needle into a target blood vessel at a target insertionsite.
 21. The system of claim 19 wherein the information compthree-dimensional coordinates, topography of the subject arm andorientation of the blood vessels.
 22. The system of claim 21 wherein thecontroller is configured to autonomously process the information fromthe plurality of sensors to locate potential insertion sites in theblood vessels of the subject, to autonomously select the targetinsertion site in the target blood vessel of the subject into which toinsert the needle, to autonomously calculate an insertion path and aninsertion angle of the needle into the target blood vessel based on thethree-dimensional coordinates and orientation of the target vessel andan orientation and a position of the needle, and to autonomously directthe needle holding tool and the robotic positioning system to insert theneedle along the insertion path and at the insertion angle into thetarget blood vessel at the target insertion site, while autonomouslytracking the position and orientation of the target insertion site inreal-time to update the insertion path and controlling a depth ofinsertion of the needle into the target blood vessel.
 23. The system ofclaim 21 wherein the one or more sensors comprise an ultrasound probe.